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FOREWORD ; 

A System for Evaluating Cbursevare for Vbcational and Technical Educatlbh 
describes the strategies used by the National Center for Research In Voca- 
tional Education to design a microcomputer courseware evaluation system that 
would meet the unique and specialized needs of vocational and technical edu- 
cation, it is anticipated that the resulting evaluation form and guide devel- 
oped in this project will better enable courseware users to assess the quality 
of mlcrococnputer courseware In the vocational and technical areas and select 
courseware of high quality. 

The contributions of technical panel participants^ pilot-test partici- 
pants, and supplemental reviewers are gratefully acknowledged. These iridivid- 
uais provided valuable feedback regarding the content of the evaluation 
system. The names of these individuals are listed in appendixes to this 
report . 

The National Center is indebted to Dr. Shirley A. Chase^ who served as 
Project Director, and to project staff members Ruth Gdrdbrij^ Program Associate, 
and Richard C. Makin, Graduate Research Associate. Other National Center 
staff who contributed to the development of the courseware evaluation system 
include Dr. Wesley E. Budke, Dr. Louise Vetter^ Dr. Steve_ Franchak ^ arid Dr. 
Robert Campbell, Research Specialists; Yvonne Berglarid, Alari Kohari, arid Oscar 
Potter, Graduate Research Associates^ and Mike Vordenberg arid Brent Miller, 
student assistants in computer programming. 

Other individuals contributing suggestipris for thie evaluation form to this 
project included Dr. Roger D. Roediger^ College of Agriculture, The Ohio State 
University; Robert Firsts Sbuth-Westerri Publlshirig Cbmpariy; arid Isaac Reed, 
Supervisor i Trade and Industrial rEducatibri, CblUmbus Public Schools. Acknowl- 
edgment is -due to Jbhri SmarJu-bf High Techiibibgy for arranging the loan of two 
Franklin micrbcbmputers arid tb the mariy irid Ivlduals and agencies^^ especially 
the curriculum cbordination centers, for sending and loaning microcomputer 
cburseware arid courseware evaluat ion forms. 

Critical reviews of the final report were provided by Dir. U>ls Hughes^ 
Chairperson, American Home Economics Association Computer Software Evaluation 
Committee; Dri Biannie Bowen, Associate Professor, College of Agriculture arid 
Home Economics, Mississippi State University; and Gale Zahniser^ Prbgram Assb- 
ciate, and Dr i J^es P. tong. Research Specialist, of the National Ceriter. 
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Recbghltlbh also is due Bettina tinkard and Michael Wbriacbtt, Program 
Associates, whb assisted with Ihe preparation of the final report ; to Stephen 
Klyce arid Ruth Waiston, typists, and Janet Ray, wbrd processor operator; and 
to Janet Kiplinger, Administrative Associate, who prbvided the final editorial 



review. 



Rbbert E. Taylor 
Executive Director 
The Natibrial Center for Research 
in Vbcational Education 
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EJ^etJTiVE SlMlARY 



The purpose of this project was to design a system for evaluating mlcro- 
cpmputer courseware for vocational and technical education. Throug^^ exten- 
sive literature. review and contacts with organizations and Individuals 
involved in courseware evaluation and use^ project staff identified arid 
acquired for review riumerdus documents pertaining to courseware evaluatibriy, 
over 100 vocational or technical education courseware products, and jpyer 40 
evaluation forms. A matrix was developed to aid in developing a preliminary 
evaluation form to be used as the basis for review and further developmerit by 
a panel of five consultants. These consultants and the five who served on a 
second panel provided valuable suggestions that were used iri refining the 
preliminary evaluation form. The second pSnel of consultants also assisted 
with the development of a guide to accompany the form. Pilot-test partici- 
pants and supplemental reviewers helped staff to further refirie the form and 
guide. Information about the evaluation system was disseminated through 
various newsletters, publications, and conferences. 

As tested and refined^ the-form^l^-organlzed ihthree parts: descriptive 
irifdrmation about the courseware, quality. criteria for rating the courseware, 
and a summary evaluation. The guide itself explairis an overall evaluation 
procedure and the details of using the form. 

Project staff also developed recommendatibris for cbritinued efforts in the 
evaluatidn df fflicrocomputer courseware for vocational arid technical education. 
The major recommendation emerging from the study i§ that a ceritralized agency 
or network provide national leadership in micrbcOTiputer courseware availabil- 
ity and evaluation for vocational and technical educatidri. 
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CHAPTER I 
INTRODUCTION 

The project A System for Evaluating Hlcrocdraputer Courseware for Voca- 
tional and Technical Education was conducted from June 1^ 1983, to February 
29, 19BA, by the National Center for Research in Vdcatidnal Education. This 
project was sponsored by the Office of Vbcatibnal and Adult Education, U.S. 
Department of Education. The purpose of the project was to develop a course- 
ware evaluation system that would focus on the specific need^lTf "vocational 
and technical education. 

Background 

One needs only to scan the headlines today to beccxne aware that we live in 
a world of computers. Computer-assisted iristruction (CAI) is of such interest 
to educators that conferences and workshops featuring information or materials 
on computers and courseware are often bverfldwing with participants. Statis- 
tics on the number of computers in schools are often out of date before they 
can be published. However, some of the latest data, collected by Market Data 
Retrieval fr^ July to September 1983 and released In October 1983^ indicate 
that more schools began using computers during the past year than in all prior; 
years combined. Now over 86 percent of senior high schools have computers. 

Vocational educators are investing valuable funds to purchase cbmputers 
with the idea of becdmihg computer literate themselves and prbviding such 
training for students. Teachers eagerly seek courseware to use on their new 
computers, only to discover that the exploration of available courseware 
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should have been conducted before the hardware was purchased. Now chey are 
finding that quality courseware to meet their instructional needs is riot 
available or may be prohibitive in cost. 

Although courseware development is progressing rapidly ±n both quantity 
and qoailty, products still are often of varying quality. There has been a 
proliferation of courseivare developed by individuals and agencies without the 
necessary expertise. Therefore, the courseware available may or may not war- 
rant its cost, so that considerable time and effort need to be expended in 
identifying, evaluating ^ and selecting courseware to meet specific vocational 
and technical education needs before scarce resources are invested. 

Currently^ evaluatibri of courseware is being conducted by many diverse 
organizations and Individual reviewers. Courseware reviews are published in 
computer magazines and databases; however, few have been found for vocational 
and technical education. Thus, potential users do not even have a basis for 
selecting courseware for preview, much less for making final decisions on 
acquisition and use. 

An evaluation sydtem that focuses on microcomputer courseware for Vbca- 
tlonai and technical education is needed to take the guesswork out of the 
vocational educator's cOUrsev:are selection. Such a system hot only will h^lp 
users of cburse^re but also can influence developers to address the specific 
needs of vocational and technical education in the products they develop. In 
.addition, the system may be used by professional reviewers in evaloatl'ng 
courseware for pabiished reviews. 

Purpose and Qbjec^iv^ 
The purpose of this project was the develbpmerit of such an evaluation 
system. The specific objectives were as follows: 

2 
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o To review the literature about the systeffls for evaluating microcomputer 
software for vbcatibhal education 

6 To adapt or design a system for evaluating microcomputer software for 
vocational education 

o To ptlot-test the system with at least ten pieces of instructional 
software 

o To disseminate information about the evaluation system through estab- 
lished National Center dissemination mechanisms 
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CHAPTER li 
PROCEDURES 

Literature Re view 

The objectives of the literature review were to identify the following^ 
lateriats and people: 

o Existing evaluation systems for microcomputer courseware 

o Criteria for courseware evaluation 

o Vocational education courseware for pilot testing 

o Potential participants for the technical panels 
'o achieve these four Object ives^ project staff conducted computer searches 
inline databases, gathered information and materials from individuals and 
rgahizatibhs identified in the searches^ and reviewed arid suminarized' the 
nforraation obtained through all sources* 

Searches were conducted of the following brilirie databases: 

o Bilingual Education Biblibgraphic Abstracts (BEBA) 

o Data Processing and Irifbrraatibn Science Cbritents (DISC) 

i 

6 Dissertatibri Abstracts International (DISS) 

b Educatibnal Resburces Irifbrraatibri Center (EfelC) 

o International Software Database 

o Microcomputer Index : 

o Online Microcomputer Software Guide and Directory (SOFT) 

o Ontario Education Resource Information Database (ONED) . 

o Resources in Computer Education (RICE) 




b School Practices Ihfbrmatibri File (SPIF) 

b Vb^£tibnal Education Program Imprbvemerit (RIVE) 

The RIVE database was especially helpful iri providing information about 
irrent projects related tb mlcrbcbmputer courseware in vocational and techni- 
il education. ; 

immary of Literature ^ 

Project staff used the resources of The Ohio State University libraries, 
le Natibnal Center's research library", and the organizational ^h§s of the 
^source and Referral Service at the Natibnal Center tb acquire printed copies 
: relevant journal articles, microfiche copies bf ERIC documents, and gen- 

relevant information. A substantial part bf the evaluation literature 
>nslsts of journal articles, conference papers arid proceedings, and mono- 
raphs that?^ address issues pertinent tb cburseware evaluation. The content of 
ich of this literature tends to be repetitive. The main recurring themes are 

need for quality courseware^ the need for an evaluation process, ^nd the 
2ed for ^n opporttmity to preview the cburseware before making a selection. 

One of the first things noted in the review bf the literature was the 
nconsistent use of terminology in the field. For this study, microcomputer 
nstructional prbgrams are referred to as ^ursew a re , aithdugh the term soft- 
are is also used in the field. Hence, both terms appear in this summary bf 
he literature. 

Five major areas for consideration in courseware evaluation are summarized 
n Micrdcdmputer Software fo^^du3.t Vocational Education: Guideliries for 
valuatibn (Stone 1983): 

o Learning objectives and task analysis 

o Appropriate use of the technology 

6 
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b Pedagogical considerations 
6 Management cons ide rat ions 
6 Dbcumehtatioh 

All or some of the above considerations are dlscussied in detail in raach of 
the evaluation literature examined by project staff. Most of the literature v 
addressed Issues pertaining to education In general. M i croco mputers in Voc 
Ed: A Decision Guide (Zahnlser. Long ^ and Nasmari 1983) notes five concerns of 
Lmpdrtarice to vocational and technical educators related to microcanputer 
zburseware: 

o Issues regarding courseware research and development are ^o^e pressing 
for vp.catibrial educatibri than for general education because vocational 
education is more specialized. 

b Technical courseware is needed for the hands-on, psychomotor activities 
typically a part of vocational education instruction. 

b New strategies for assessing computer-assiste Instruction are needed tc 
meet the training and retraining needs of adults. ^ 

o Fewer commercial programs are available for vocational education course- 
' ware because the market is much smaller than the market for general 
education. 

o Vocational educators may need to develop courseware to ensure that the 
curriculum reflects the requirements of local employers. 

A variety of other publications were acquired for review. Publicatibhs 

c 

directed at national audiences include the National Cbuhcil of Teachers bf 
Mathematics' Guidelines for Evaluating Cbmputerized Instruc tional Materl als^ 
(Heck^ Johnson, and Kahsky 1931). Jburrials such as Infoworl d , EdQcat l o jxa± 
technology ^ and The Cbmputihg Teacher regularly publish narrative Reviews of 
courseware. For those wishing to compare evaluatibns bf a particular product, 
rhe Digest bf Sbftware Ro views ; Educ: a4^1ofl provides abstracts of published 
svaluatibiis. 
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Evalaa t ton of Educational Software; A Guide 'to Guides (Jbries and Vaughan 
1983) provides a cbtnpllatlbh of ten different evaluatldi? forms and a brief 
description of the brgarilzatlbris that developed them, along with three com- 
pleted sample evaluations. The introductory chapter, "Evaluating the Evalua- 
tion Schemes," was written by Henry F. Olds^ a rioted specialist in the fields 
and includes critical comments on several of the major evaluation systems In 
current use and offers suggestions for the review process. 

ether publications and ihfbrmatibri were available from organizations that 

have existing" evaluatibh systems that are directed at national audiences. The 

*-j ' ■ ' _ _ 

materials from these organizatibris were alsb reviewed. These organizations 

i ----- ------- ----- — ^-'-L 

and the materials they putfllsh arid distribute ate as' follows : 

o CONDUIT distributes cburseware that has passed eONDUIT's own review 
'process arid publishes courseware descriptions in Pipeline ^ its biannual 
journal. 

o Educational Insights, publishes Cour B eWar ^ Report Card , a Journal of 
reviews that is available Iri twb editions (elementary and secondary). 

b Educatlbrial Products Iit fortaatlon Exch a nge and eonsimers Union (EPIE/CU) 
pnKI-fQh rPv-IPWR of commercial courseware in Mtcro-eoarseware PRO/FILES ^ 
which are available by subscriptibri. Sample PRO/FltES alsb^ are Included 
in Microgram, the EPIE newsletter. 

o National Educatlbri Asso ciation (NEA )- offers on subscription a catalbg bf 
NEA Teacher Certified software and publishes a series bf three guides 
that detail the criteria used to evaluate software submlsslbris. 

b Northwest R egional Educational Laboratory tNWREL) Includes software 
evaluations in the RtfiE database and publishes MlcrbSIFT reviews in The 
Compattn^ Teacher and other journals. NWREL developed the MlcroSIFT 
Evaiuator's Guide (Internatlbrial Cburicll fbr Cbmputers in Education 
1983). 

Also reviewed were three projects described iri the RIVE database. These 



projects address the courseware evaluation needs of vbcatlbrial educators. 
They are as follows: 
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o Xiietttlf Icatldit and Ev^luat4cm^^ Co mputer Softwanfe in Home Economics — 
Conducted from January through June 1983 (Hovis &nd Bloom 1983) at 
Indiana University of Pennsylvania, the project developed ah evaluation 
form for use in five areas of home economics. 

b Software Reviews Learning Center at the Bureau of Vocatiohal 

Education — Conducted from January "t^hrough June 1983 by the Kentucky 
State Department of Education, this project was concerned with devel- 
oping software evaluation procedures for implementation at the ^learning 
center • 

o Development of an Evjluatiye Instrumen t for Comp gtei^^rograms with 
Application in Vbcatfonal Education — Oregon State University conducted 
this project from October 1982 through February 1983^ An evalaation 
Instrument was developed and tested with courseware related to voca- 
tional and technical education subject matter. 

Computer searches of the RICE and RiVE databases were ithe most useful in 
tdentifying vocational and technical education courseware. In addition to 
iescriptidns of courseware, RICE provided a list of producers who could be 
contacted for catalogs and additional titles. RIVE provided irifbrmatibri on 
courseware that was developed with Federal and ^state fUnds administered 
through the state departments of vocat&nal education. 

A compilation by Rode^stein and Lambert (1983) provides a listing of 
courseware programs for vocational education brgahized by vocational area. 
The programs are recommended as "quality prbducts" and "worth examining." 
Fhey have been "reviewed but not formally evaluated." The 1983 Educational 
Software Preview Guide (Educational Sbftware Evaluation Consortium 1983) 
includes business education^ industrial arts^ and home economics programs, 
the programs listed in this guide are "recommended for preview." 

Resources for small business applications are the focus of Microcomputers; 
Vocational Trainin g for Small BuB i nes s M anagement (Heath and Camp 1983). 
Included in "this monograph are detailed descriptions of selected instructional 
pa.ckages for teaching concepts and skills in small business management, along 
with listings and descriptions of business applications packages. 

9 
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The Mimtesbta Educatiohai Computing Coriisortluffl (mee) (n.d.) includes 
vocational education prbgraffis in ils publication MECC Instructional Compottng. 
Catalog . Vbcatibnai education courseware also is included in the catalogs of 
the various commercial developers. (A selected listing of the courseware 
acquired and reviewed by project staff is included in appendix E of this 
report;) 

Cbritact& 

Informatibn gathered in the coipnter searches enabled project staff to 
formulate a list of organizations and individuals to use as contacts. Con- 
tacts were made with organizations such as Ibcal secondary schools, intermedi 
ate scho^TTiltricts, curriculum resource centers, curriculum coordination 
centers, technical schools, colleges and universities, state depar tSents of 
education, and other vocational and technical education agencies. Names of 
these brganizations and of individuals were acquired through the computer 
searches, through the initial cbntacts, and through responses generated by 
requests for information and courseware products. Through these contacts, 
over Idd vocational and technical education courseware products were identi- 
fied. Most of this Cburseware was compatible with the Apple lie micrb- 
compiiter. Therefbre, that hardware was leased for prbject" staff tb use in 
reviewing the acquired prbducta. in addition, over 40 cbUrseware evaluation 
forms cbvering various areas and levels of education were identified and 
acquired.; However, few of these evaluation forms related specifically to 
vbcatibnai and technical, education and many of the fbrms did not evaluate the 
newer features of available cburseware. 

Information concerning how practitioners select courseware was also 
obtained. This information implied that while some practitioners have 
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stablished criteria they use in courseware seiectibni many simply make selec- 
ibris bri the r^cbmTiiendat ion of others. ,Ail of the InfbrmaUioh gathered 
hrough contacts supported the belief Chat there is ^ real need in the field 
:br a system tb evaluate fflicrocomputer courserare that is specific tb the 
Leeds of vocational and technical education. 

Lgency Visits 

Shirley A. Chase visited MECC on August 9, 1983. The purpose of the visit 
^s to become familiar with the activities and services of MECe and to discuss 
[ational Center prbject effbrts \Hth the MECC staff. Karen Jostad, a MECC 
;taff raeonber^ reviewed the cburseware evaluation form, then in its early 
;tages of development^ and made cbmmehts and suggestions. A tour of the 
acility and visits with support staff were also a part of the visit. 

Dr. Chase attended the second annual Microcomputers and High Technology in 

rbcatibrial Educatibri Cbnfererice sponsored by the Vocational Studies Center in 

^ _ __ _ _ _ ___ _ _ : 

fadisbn, Wiscbnsin, August 10-12, 1983. The wide array of conference ^esen- 

latibns and exhibits and the number of contacts available during this cbnfer- 

mce provided an exceptional opportunity to learn the state of.. the art of 

aicrocomputers for instructional purposes in vocational and technical 

education. ^ ' 

Ruth Gordon visited the Oswego County Board of Cooperative Educational 

Services (BOCES) in Oswego, New York, on November 7, 1983. BOCES develbps 

rourseware for slow leatners who have trbuble reading regular materials. Stu- 

lerits use the courseware individually or in small groups. Kathy Finnerty, 

JOCES staff member i reviewed the cburseware evaluation form and offered sug- 

^estibris fbr develbjping a useable evaluation form. 

, 11 • 
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Ms. Gbrdba also visited EPIE/CU Iri New York City on November 9^ 1983. 
Staff m^hei Ellen Blalo described the EPIE/CH procedures fbr, cburseware 
evaluatlbrii prsparatlbri of the Miero^HCoarseware PROFILES ., and liifbrmatlbn 
dissemination. She also offered suggestions for the courseware evaluation 
system being developed. * 

Dr. Chase attended the MECC Educational Computing Conference bri November 
21-22, 1983, in Mlnneapblls, Minnesota. This conference provided an array of ^ 
activities presenting the state of the art of microcomputers in all of educa- 
tion. The newest features of hardware and courseware were highlighted, giving 
the basis for adapting fhe National Center's courseware evaluation system. 

Dr; Chase attended the pre-AVA Conference "The Use Of Computers in Voca- 
tional Instruction" on December 1, 1983. The types of courseware available 
and how to evaluate it Were the main topics bf interest. Review of the mate- 
rials made it evident that the National Center's System for Evaluating Micro- 
computer courseware in Vocational and Technical Education is more cbmprehen- 
sive: and specific to the field than any bf those displayed or reviewed during 
the conference. 

Richard C. Makin visited NEA Educational Computer Service in Bethesda, 
Maryland, o;i December 12, 1983. The purpose of the visit was to gain knowl- 
edge of the NEA software evaluation system and to obtain comments from NEA on 
the courseware evaluation system for vocational and technical education being 
developed at the Natlbhal Center. On behalf of NEA, Carbl Trawlck exp3atned 
that the primary cbncern of NEA is the communication bf cburseware evaluation 
standards that are applicable on a nationwide basis. NEA has published its 
fLrst major catalbg bf microcomputer courseware products, but the catalog does 
not contain any listings for vocational and technical education. Interest and 
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support were expressed for the National Center's courseware evaluation system, 
especially for the sections that related specifically to vocational and tech- 
nical educatibhi an area that NEA has not yet addressed but is interested in 
exploring in the future. 

NEA has developed three documents that make up a guide to the software 
assessment procedure to be used in software evaluation. These documents are 
available to educational users and the general public who want to know the 
criteria currently used to evaluate NEA Teacher Certified software (National 
Education Association 1983a ^ 1983b^ and 1983c). 

Shirley A. Chase visited NWREL February 15^ 1984. The visit included 

meetings with Dr. Thomas Owens and other staff members. The RiCE database and 

other services of NWREL were explained. Collaborative efforts between the 

National Center arid NWREL in the areas of microcomputer courseware evaluation 

and brilirie databases were explored. 

This same trip also included a visit to the Vocational Education Division 

> 

of the State Department of Education; Salon, Oregon. Wanda Mbhthey hosted Dr. 
Chase's visit on February 16-17 to the state department^ which provides 
.funding and support to many courseware development projects in the state; to 
Lane Community College, at Eugene; and to Oregon State University at 
Corvallis. Activities in microcomputer courseware develbpmerit arid evaluation 
were discussed and observed at these institutions. 

During each of the visits^ visibility was given to the National Center's 
courseware evaluation system. Suggestibris fbr its dissemination and use were 
secured. ' 
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Related Activities ; 

The literature review also resulted in the idehtlf Icitibh of persons 
actively involved iti the use of mtcrocbmputers for classrbdtn instruction. 
This irifdrmation was useful in compiling a list of potential tnembers for the 
two technical panels. During the course of the project, a fifth objective was 
tedded for the literature review: to provide vocational and technical educa- 
tors with comprehensive resources about courseware and evaluation and avail- 
ability of vocational and technical education courseware. This objective was 
added because of the numerous requests for such information received by pro- 
ject staff. Therefore^ the literature review, which was originally planned 
for only the first 2 months of the project, was continued by staff through 
updates of online searches of the ERie, RIVE, and RICE databases and continued 
review of Mterlals until the preparation of the final report. . 

Evaluation System Ad aptati o n and Bestgn 
Lopment of the Evaluatibh Form 



Evaluation forms Were cblleUed through a review of the liter-atuEe, 
through personal cbntacts, and lihrough a request for such fbrms in the project 
profile, electronic newsletters , Und the National Center's Cehtergram. All 
forms were then studied to deter£?.ne If an existing fbrm could be' used , either 



Intact br revised, to meet the heeds of vocatibhal and technical education. 



Since no fbrm was discovered of sufficient scbpetb cover all aspects of voca- 
tlbnal and technical education, project staff undertook to develop an original 
form through a cbmbinatibh bf apprbacttes. 

FiHr, a matrix l^Ss develbpd^^^u^^^^ tm^indices-of" evaluation criteria and 
luation-forms By using this matrik, staff quickly determined which 
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evaluation criteria were present in given evaluatlbh forms. If a particular 
criterion was present In a large percentage of the forms, that criterion was 
included in the initial draft of the preliminary evaluation form- In some 
cases, project staff judged as important evaluative criteria that were preseat 
in only a small percentage of the evaluation forms; these too were included in 
the* initial draft of the preliminary evaluation form. After compiling a list 
of evaluation criteria, project staff grouped criteria of a similar nature 
Into sections* The sections were based on those present in the collected 
evaluation forms. 

The criteria were next reviewed by individuals ^th a background in micro- 
computers in education. Their input was particularly important in regard to 
the technical aspects of courseware evaluation. Additions or deletions were 
made, to the evaluation criteria on the basis of the recommendations of these 
individuals. Project staff also contributed criteria identified through 
actual review of vocational and technical education courseware. 

The evaluative criteria identified through these three sources were then 
revised as needed for conciseness and clarity. Selected National Center staff 
members then reviewed the evaluation form prior to the cbriveriing of Panel i. 
Recommendations were incorporated as appropriate. In addition to these eval- 
uative criteria, descriptive items were concurrently developed. These items 
organized a body of information about the courseware: identification, system 
requirements, instructional setting, and a general description of the^program. 
They were grouped in a separate part of the microcomputer cotirseware evalua- 
tion fbrra^ Part A. As presented for review, to Panel I participants ^ the 
evaluative criteria (now Part B of the form) were grouped in the foD^vihg 



sections: ^ v 
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b subject matter content ^ 
o ins true tib rial design 

--subject matter presentation ^ 

— technical presentation 

— teacher use 

; — student use ^ 
— feedback 

— evaluation ^ 
— personal/ social development ^ 
— vbcatidnal development ^ 
Also compiled for review by Panel I were 22 response formats. From these, 
one was to be selected and used in conjunction with the evaluative criteria in 
Part B. The basic variations included the following; 
o yes, no, open-ended description 
o yes, no, not applicable 
o tow to high rating 

o excellent, very good^ gbod^ poor, very poor 

o 1-5 tikert scale ' 
o weighting and Llkett scale ±n combination 

Nuie^ous versions of the above response formats were found on aclstlng 
forms and In the literature, accounting for the total number of 22 response 
formats collected. 

panel I Review 

Panel I was composed of five persons both from general and Worn vocational- 
and technical education. Ml panel members had experience in developing 
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evaluation instruments or expertise in the ralcrbcanputer- field. -PrlbT-t6~the" 
meeting, each panelist was sent a packet of materials Including a project pro- 
file and a tentative agenda (see appendix A). The panel met for 2 days^ oh 
August 24 and 25, 1983. The purpose of the panel was to assist jprb^ect staff 
in the farther development of the preliminary courseware evaluation system. 

On August 2A, the panel worked primarily on Part B of the evaluation form. 
Each participant was asked to rate each criterion in Part B bh a scale of 0-3 
(3 = essential, 2 = important, 1 =» of minor importance^ 0 = nbt needed). This 
activity and the ensuing staff-directed discussibn led to the addition, dele- 
tion, or revision of many of the criteria. Majbr changes included the recom- 
mendation that a summary (Part C) be added arid that ari application programs 
section be added to Part B. 

Revisions included the follbwing: 

o Integrate subject matter content and subject matter preseritatidri 
sections 

o Expand the technical pr esentat;ibn sectibn 

o ehange the title of the teacher use section tb "Teacher Manual" 
o Delete items in the evaluation sectibn 

Prior to adjournment for the day, a response format was selected from 
among 22 alternative fbrmats. The format selected requires the user to check 
YES, SOMEWHAT, NO^ or N/A for each ::riterion. * Panel members suggested the 
incbrpbratibri bf a C^^MTS column into the response format - Panel members 
snrbngly recommended that open-ended questions al5o be incorporated iritb the 
response format. 

On August 25, prbject staff directed pariel effbrta toward two major objec- 
tives. The first bbjective waf' tb study arid revise Part A of the evaluation 
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Jo:mj:zrzSin_ce_mucl^^^^ therein is factual ^ many changes dealt 

with semantics. Revisions Included the foilbwing: 

6 Separate the availability Itans In the identificatlori sectibh and 
include them In a separate availabj^^ii ty section 

o Delete the general description of the program 
In general, Part A of the courseware evaluation form was accepted as pro- 
posed Iri the preliminary form; 

The second objective of the day was the actual use of the courseware eval- 

' ■%» ■ 

uatlon fortr. by the panel members. The Intent of this "hands-on" experience 
was to raise questions and Issues regarding the ease of use, clarity, and use- 
fulrier;s of the courseware evaluation form. Each member of the panel reviewed 
courseware and applied the evaluation form to that product- Courseware was 
reviewed In the areas of Industrial arts, health services, home economics, and 
trade and Industrial education. 

following the reviews, panel members and project staff collectively dis- 
cussed concerns and problems related to the courseware evaluation form. Revl- 
slons were made as needed- Prior to aajduritment of the panel, project staff 
requested suggestions for the guide that would accompany and be used with the 
evaluation form. It was decided that specific terms should be Identified and 
defined In a glossary- Mention was made that the guide should Include 
recommendations on how to USe the evaluation form and contain a disclaimer 
regarding the ultimate purpose of the evaluation system. 

Panel I members were asked for suggestions on how to improve the review 

■ _% _ : _ . ____ 

process for use with Panel II. The following suggestions were given: 

6 Provide an pppbrtunity for participants to review the Courseware 
evaluation form prior to the panel meeting 

o Provide an opportunity for participants to review the evaluation form 
listing arid bibliography prior to the panel meetirig 
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d Provide a detailed expiariatibri of panel objectives arid purposes prior to 
the ipaheT"mee t iiig 

After recommeridatibris from Panel I weire incorporated, the evaluation form 
consisted of the following: 
o Part A: 

— identification 

— system requirements 
— instructional setting 
— availability 

b Part B: 

Sectibn ' Nu mber of Criteria 

— subject matter _ _ __ ^? 

— technical presentation 16 

— teacher's manual 2 

— student use ' 8 

— feedback 9 

— e\7aluation 5 
— personal/ social development 7 
— vocational development 6 



Panel II Review 

L _ ___'*^ " '_ " "_ 

Panel li was convened September 21-22, 1983, 'at the Natibhal Center, This 
panel was __composM.of _f Ive vocational and technical educatbrs wfc had experi- 
ence in development or evaluation of tnicrdcbmputer courseware. These persons 
were also potential users of the completed cbutseware evaluatibri system, so 
they had a special interest in its develbpmetit . 

After Panel II participants were selected , they were sent a packet of 
materials (see appendix B) , including a letter of confirmation, a tentative 
agenda for their meeting, and a project profile. Later, a second letter was 
mailed to keep participants informed and to alert them to pick up a packet of 
informational materials, including the evaluation fbrmi on arrival at their 
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motel the evening prior to the meeting. Thii gave participants an opportunity 
CO scan the materials they would be working with the following day. 

The meeting was opened With an orientation session. Then Panel II partic- 
tpints were asked to use the evaluation form to review cbursev^re. Two 
Franklin computers on loan and the Apple lie leased for the project were pro- 
vided so that no more than two participants worked at a computer at one time. 
After the courseware review, a session was held tO solicit general comments 
and suggestions on the evaluation form. 

Considerable time was spent reviewing Part B of the form, Revised as a 
result of Panel I recommendations. The relevance of each criterion was dis" 
ciissed by- the group using the following considerations: 

o Importance to courseware evaluation 
' o AppllcabillS to vocational and technical education 

o Language/ terminology used 

o Coverage of important issues ^ 

Part B of the form was further refined by eliminating some criteria, 
revising others, and making some ^ddltlons . After this refinement process, 
two new sections of the evaluatlon"^form Were developed: documentation and 
application.^ Panel II gave support to retaining the section on vocational 
development, viewing those criteria as critical to vocational and technical 
education curricula; 

The second day was opeiied ' v*ith a session to review more courseware, this 
tlmi using the refined version of the evaluation form completed the previous 
day. Subsequently, Panel II members made their final revisions to the course 
ware evaluation form* 



Each Panel II meffiber was asked to prepare written recommendations on the 
courserare evaluation form and its use. A summary of the recommeridatidns 
follows: 

o Add an overall recommendatibh Item and a listing of sections as a 
one-page summary to the evaluation form 

p Provide vocational and technical education state department personnel, 
teacher educators, and curriculum centers the more detailed > comprehen- 
sive courseware -Evaluation form and develop a short version of the 
courseware evaluation form for use by classroom teachers 

o improve the evaluation form layout 

o efiahge the application section title to "Applicatibri Programs Only" 

o Strengthen the "Application Programs Only" section and move it to the 
end following the vocational development section 

o Strengthen the vocational development section 
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o Add a COMMENTS column on the right side of the page so comnients_c''an be 
^ made for each criterion included in Part B of the evaluation form 

o Conduct field testing ^ *^ . ^ . ' 

o Disseminate the courseware evaluation system through curriculum centers 

o Provide information on the courseware evaluation system through existing 
National Center publications ' 



o Direct promotion to the following target audiences: classroom teachers, 
local administrators, state department personnel', teacher educators, and 
curriculum developers and reviewers 

b Distribute the evaluation system in conjunctldn with a national cieai 
ihghbuse to ^prevent: the duplication of courseware evaluations in voca- 
tional arid techriical education 

d Conduct courseware evaluations and disseminate the reviews through the 
Natidrial Network for Curriculum Coordination for Vocational and 
Techriical Education (NNCCVTE) and the Vocational Education Curriculum 
Materials (VECM) database 

o Conduct workshops on the courseware evaluation system at vocational edu- 
cation conferences 

a Train individuals (e.g., state liaison representatives) to develop dis- 
semination procedures that would be appropriate to their states dr 
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regions so that they could in turn train others in Che ase of the 
courseware evaluation system ; 

b Obtain assistance from state directors t;b provide mtntworkshops thtough- 
out states to inservice teachers on the courseware evaluation system 

b Request that colleges and universities USe the form with graduate 
classes in vocational and technical education 

0 Enlist teacher educators to prbvide preservice vocational and" technical 
educattbn teachers with training on Che use of the courseware evaluatior 
system 

Panel II members were divided into two i^bups to work in developing por- 
tions of the guide to accompany the courseware evaluation form. They worked 
on the item description portion of the guide that is cbrrelated to Part B of 
the courseware evaluatibn form. These developmental efforts were used later 
by pro ject staff in'the final developnient of the guide. 

Following the incbrporation of Panel II recommendatibris , National Center 
staff members reviewed the courseware evaluation form.. The changes suggested 
and incbrpbrat^d were as follows: * ^ 

o Change the title Of the vocational development section in Part B to 
"Work Behaviors" 

o Use a mbre. cbrisistent format in Part A 

o Include specific directions for each section in Part A 

o Include preview pblicy 'in the availability sectibn bf~ Part A 

b Include a section summary ±n Part C^ 

b Change" the title of the evaluation summary in Part C to "Final 
Rec bmme hda t io n" 

( Pil ot Test 



Trial use of the evaluation system was conducted with the assistance Of 
three grbups of people. These were National Center staf f members , vocationa3 
practitioners, and . curriculum and resource people. Following the 
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tncorporatidn of recdtnmendat lonis from Panel II, the evaluation system was 
first used on a trial basis with National Center personnel, this occurred oh 
November 1 and 2, 1983. The purpose of this trial was to establish procedures 
for reviewing vocational and technical education coursewa:re and to obtain 
feedback regarding the clarity and feasibility of the evaluation system. Oh 
the basis of information found in the literature, several alternative 
sequences of activites for courseware evaluation were devised and applied. 
Following the internal trials with the evaluation system^ project staff and 
participants then discussed the effectiveness of these alternatives. This 
resulted in the adoption of a review process to be used during the second 
phase"^" the Trial with vocational and technical teacher reviewers. 

The courseware evaluation system was next used with local vbcatibnal prac- 
titioners in agriculture^ business and of f ice^education, health services, home 
economics, industrial arts^ and trade and industrial education. The back- 
ground of the teachers in both using the micrbcbmputer and evaluating course- 
ware was quite diverse. This prbved-behef icial in that different concerns and 
suggestibhSi which were based oh the individuals' different levels of experi- 
ence, were expressed. The purpbse bf the trials was again to obtain feedback 
regarding the ease bf use bf the system and clarity of both the evaluation 
form and guide and to secure suggestions for improving the evaluation system. 
Staff members sought practitioners' suggestic/ns for revision of the review 
process resulting from the internal trial with National Center staff. For the 
most part, the practitioners were receptive to the review strategy employed. 
This trial-use portion bf the project was conducted individually vd:Th"teacher 
reviewers between November 8 and November 30, 1983, at the National Center. 
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The process during the trial use was as follows: participants were firiS 
given a brief orientation regarding the scope and purpose of the project. 
Emphasis was placed on the purpose of the trial--i.e. , evaluation of the eva 
uation system. If necessary, participants also were given an introduction t 
and vdemonstration of the Apple lie computer. Following the orientation, par 
tlclp'ants were encouraged to review the evaluation system and any documenta- 
tion accompanying the courseware. The teachers then reviewed their program 
twice, as both good and poor students. The evaluation form was then com- 
pleted, with teachers allowed to refer back to parts of the pr6gram when 
necessary. The last phase of the process consisted of both a written and a 
oral evaluation of the evaluation system. 

The discussions between project Staff and participants at the close of t 
process were of particular Value. Trial use of the evaluation system on an 
individual basis averaged 2 to 3 hours. As a whole, participants . in the 
trials were receptive to the evaluation system and shared a consensus that t 
system would be of value to vocational and technical educators. Comments ar 
suggestiuris were incorporated into the courseware evaluation system when 
appropriate . Major recommendations -from the practitioners included the 
following : 

6 Revise the format 

o Expand the application section . 

o Retain the wdrk behaviors section ^ 

o Retain the detailed, comprehensive evaluation form 

o Clarify and expand Part A of the evaluation form (courseware feature 
items in particular) 

The third group of individuals to make trial use of the evaiua^ibh syst 

was curriculum and resource people attending the Sixth Nationwide Vocatlona 
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Education Disseminatiou Conference at the National Center. This aspect of the 
trial use was conducted on November 16, 1983. Participants registiered for the 
evaluation session as part of the conference. There was again great diversity 
in the participants' vocational background arid previous experience with 
courseware evaluation and use of microcomputers; 

The review process that was used with the vocational arid technical practi- 
tidrters was agairi used with the resburc'- arid curriculum specialists- Recoffl"" 
meridatibris made by these individuals are sumtaarized : 

b Revise the techriical preseritatibri sectibri — 

6 Retain the work behaviors sectibri 

b Clarify the prbcedure tb fbllbw in evaluating courseware 
During arid fbllbwirig the pilbt test> twb representatives bf commercial 
cburseware develbpment firms visited with project staff to demonstrate course- 
ware: Robert First, South-Western Pabiishing Company; and Chris Sakelaris, 
Borg Warner Educational Systems Division. These demonstrations provided 
project staff with greater insight into the state of the art in commercial 
courseware development. 

Dissemination of Project Information 

Information: about the project was disseminated through a bne^page project 
prbf ile and through news releases that included requests fbr help in Ibcating 
evaluation forms and vbcatibrial arid techriical educatibri cburseware. These 
releases appeared iri the fbllbwirig publidatibris : 

b Natibrial Pbstsecbridary^ Alliarice Ei^s^and^^ ieces , July ' 1983 

b Career— P ^rinin g^ ^nd Adult Development Newslette r^ , vol. 5, no. 10, 
Octbber 1983 

b Career Educatibri New# , vbl. XII, rib. 19, November 1, 1983 
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National Associatlbh of Secondary School Principals 
6 American Teacher , vol. 68, no. 4, Deceiber 1983/January 1984 
The same type of release was included in two electronic newsletters, Hlg^ 
TecH for Pd&^s^aondary Edixcators and Vocational Education Newsletters (VOCN), 
arid released through the electronic mail system. It also appeared in the June 
29, 1983, issue of RBx Monthly Memo , compiled by the Far West Laboratory for 
Educational Research and Development for distributibri to members of the 
Research and Development Exchange. 

These news releases resulted in many respbrises that provided contacts, 
courseware products, and evaluation forms for prb^ect staff to reviews 
Requests for information and materials on the project also were received. 
An inforiational article was published in the October 1983 issue of the 

Natibrial Center* s Centergram . This article also led i:o many requests from the 

f 

field for inforiation and materials on the availability and evaluatiSn of 
vocational and technical education courseware. The large number of requests 
and other responses generated by the articles conf itttied the current need for a 
courseware evaiaation system related to Vocational and technical educatibri. 

Information about the project was also disseminated at conferences and 
workshops. Shirley A. Chase attended a workshop on microcomputers spbnsbred 
by the Ohio Department of Education for vocational directors an^ supervisors 
on August 2, 1983. The session Attended was for home economics supervisors. 
The presenter was Nancy Dillon, from Strictly Sbftwarej Inc., wHb spoke on 
using micrbcbfflputers for instruction. Dr. Chase irifbrmed the group about the 



National Center study on microcbu-Suter cours_^waxe-^evalu^ibn- 

Ruth Gordbn presented "Hands-on Experience in Reviewing Home Ec^ndmics^ 
Software" at a joint meeting of the franklin County Hbme Econdmics Association 
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and District D of the Ohio Home Economics Association on September 29, 1983. 
This activity provided visibility for. the National Center's courseware evalu- 
ation system* 

Dr. Chase also gave presentations on computer courseware evaluation to the 
following groups at the itarierican Vocational Association (AVA) Conventlbri 
December 4-6, 1983: 

o Home Economics Teacher Educators 

o Vocational Instructional Materials (ViM) 

o American Vocational Education Research Association (AVERAJ session 
(prepared paper) 

Visibility vras given the project at the Sixth Nationwide Vbcatibrial 
Education Diosemination Conference on November 16-17^ 1983. The two workshop 
sessions were titled as follows: 

b "Evaluating Vbc Ed CbursewaxB^^ 

o "Getting Acquainted with Voc Ed Courseware" 

Agency visits by prbject staff as meritibried previously in this report 
promoted and gave visibility tb the study. 

Descripttion bf Micrbcbmputer^ Coursew a re 
E valuatibri Fbirm and Guide 

The final courseware evaluatibh system fbr vbcatidnal and technical educa- 
tion consists of two cbmpbhehts. These are the courseware evaluation ^pm and 
the courseware evaluation guidd. 

The courseware evaluation form contains three parts ^ labeled A, B, and C. 
Part A organizes descriptive information regarding the courseware that is 
being evaluated. The five subcomponents of Part A are identif icatibn ^ hard- 
ware requiroients, program features, instructional setting^ and availability. 
The user ccmplet : as much of the requested ihfbnnatibh as possible fol'lowing 
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a ptellffilnary review 8f the courseware. Cbmpietion of Part A cin serve as ah 
Initial screening device for courseware use; By comparing the descriptive 
infottnation sappiied in Part A with instructional needs, raanjr users will be 
able to determine whether the courseware meets basic requirements—hardware, 
for exaiple--before going on to the lengthie^. Sore detailed evaluation in 
Part B. Other users^ such as professional reviewers, will find Part A a con- 
venient means of organizing this. basic information for further reference. 

Part B of the courseware evaluation form consists of sections of evalu- 
ative criteria. SeVenty-four criteria are grouped into eight sections. The 
sections and their corresponding number of criteria are as follows: 
section NumbeJ^43l Criteria 

b Subject matter 

b Technical presentation \% 

o Student interaction . ^^11^ 

b Ptbgr^ interaction ^ 

b Student evaluation 9 : 

b Dbclitnentation 

b Wbrk behaVibrs ,^ 
o Application programs 

The user is instructed to check the appropriate response for each criter- 
ibri (YES, SOMEWHAT, NO, N/A) and to write explanatbry comments when needed. 
It is impbrtant to note that not all sections wOuld be applicable to all 
cburseware. 

Part e of the cburseware eValUatibfl form provides for a summary rating of 
the courseware being evaluated. Users are asked to identify strengths and 
weaknesses of the program as well as potential uses in an instructional 
setting. Sectibns bf evaluation criteria are rated as a whble in the same 
Sanner as the previously discussed individual criteria. The last subcomponent 
of Part C is a final recommendation, with explanation* of the cburseware being 
reviewed. 
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The courseware evaluation guide was written to assist users of the course- 
ware evaluation form. Included In the glilde are a brief description arid | * 



iexpla nation of the courseware evaluation fonui a suggested procedure for lisi 
the courseware evaluation systeadi a glossary of terms ^ and a brief explanation 
of every Item contained In the actual cbuiTseware evaluation form. ^ 

The description and explanation of the 'evaluation form briefly describe 
each part of the courseware evaluation form and then go on to explain how each 
part should be completed. A procedure Is suggested for applying the course- 
ware evaluation syston to vocational and technical education courseware. The 
procedure should be appropriate for most of the target audiences, with profes- 
sional reviewers being encouraged to modify the procedure to satisfy any spe- 
cial needs* The suggested evaluation procedure Is also presented schemati- 
cally in a flowchart. The glossary is brief by design and contains only 12 
terras that had taken on a specialized meaning or were widely used in the 
courseware evaluation system. The largest portion of the courseware evaluation 
guide is the item explanations. Each item in the three parts of the course- 
ware evaluation form, including the evaluation criteria, is explained for 
users* reference. 

Although an attempt was made to develop the courseware evaluation form in 
a straightforward, understandable style, beginning coursev^re evaluatbrs in 
particular should find the courseware evaluation guide beneficial In clari- 
fying any Items or criteria not understood. Experienced courseware evaluatbrs 
may heed only refer to the courseware evaluation guide for clarif icatibh bf a 
few of the evaluation criteria* The courseware evaluation system in its 
entirety is presented in chapter 3. 
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CHAPTER III 



A SYSTEM FOR EVALUATING MICROCOMPUTER 

COURSEWARE FOR VOCATIONAL AND TECHNICAL EDUCATION 
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KICROCOMPUTER COURSEWARE EVAtUATiON FORM 

NOTE: If you are usina this form for the Evaluatbr _ . — . 

first time, read the instructions In the Position 

accompanying Microcomputer Coursevzare 

Evaluation Guide. ^^^^ 

Part A; Description 

In the following sections^ record descriptive information about the 
courseware that you are evaluating. 
T. _IDENTiFieATibN • 

Program Title — ■ — 

Series Title - . ^ ~ 

Vocational Area(s) , 

Subject Area(s) ! ~ . . .. , — 

Topic (s) — 

Developing Agency - 

Street or P.O. Box -- : . — ■. • 

City State Zip Phone , 

Author (s) y ■ ■ — - 

Programmer (sV^ ■ — - — — : 

II. . HARDWARE REQUIREMENTS 

Microcomputer* - 



(brand/model) 

K Memory Required - -— .. ' ■ - -- 

(number) 

Medium of Transfer (include number of each): 

Tape cajssette _5H' Flexible disk -Other -_ _ 

ROM cartridge 8" Flexible disk ~ ^ 

(specify) 

Programming Language - DOS Specifications 

- _ - - * ' ^* . . ^ ■ 

Other Specifications . 

Peripherals Needed (check all that apply)- 

^Goior monitor Modem _-^Clock,, ^ 

' ^One disk drive ^Mouse V ideo disk_ 

Two disk drives Printer T ouch screen^ 

Plotter ^Graphics tablet T en-key number pad 

Game paddle (s) Li^ht peri ^Other 



Joys tick (s) ^Voice/sourid instrument . — 

—J fsnecif y) 

* NOTE: Provide the above information for any additional hardware' on 

which this program can be used- ' - . ^ 
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III. PROGRAM FEATURES (check all that apply) : ^ 

Network version provided P rogram protected 

• "Multiple, copies required Bata disk needed 

P rocram can be mqdif led ^Field-test data available 

IV. INSTRUCTIONAL SETTING 

Program mode (check all that apply) ^ - ^ ^ 

Application Educational gaming ^Tutorial 

Drill and practice Stmuiation ^Other 



(specify) 



Student Target Population (check all that apply): ^ 

Regular ^Handicapped. —^Bilingual 

D isadvantaged ^Limited English G ifted 

Grade Level (check ail that apply): „i ^ 

9-10 13-14 ^ ^Higher Ed 

"7-8 11-12 Adult 



Instructional Grouping (check all that apply): 

Individual _ , 

~^mall group (up to 4) competitive interaction 

"Large group (4 or more) cooperative interaction 



Prerequisite Student Skills (specify)__ 

Accompanying Materials (specify types) 
Ddcdmentat ion 



Student support materials_ 
Teacher support materials_ 

Correlated materials_ 



Estimated Time for Use_^ 



V. AVAILABILITY 
Free: 



Sale 



(copies) 

Loan 



Rent 



(time) 

Duplication (requestor supplies disk) 



(time) 



Gopyright Restrictions (explain)_ 

Back-^-up Policy (explain) 

Preview Policy (explain) 

Update Policy (explain) 



Contact 



Street or P.O. Box^ 
City— 



State ZiP_ Phbrie_ 



34 



39 



Part B: Evaluation CMrarfeer-lna 



indicate the applicabili ty of each section, to thia courseware being 

evaluated by checking either/' ^A'' (applicable) or " ^N/A" (riot applicable); 

If a section Is not applicable^ proceed to ^the next section. If a section 
is applicable^ check the column that indicates how well the courseware 
meets each criterion. Include any cdnmients. 





YES 


SOME- 
WHAT 


NO 


N/A 


COMMENTS 


I. SUBJECT MATTER A N/A 




1. Subject matter has educational value. 












2. Student objectives are stated. 












3 . Subj ect matter is accurate . 












4, Subject matter is logically presented. 


















5. Subject matter is free of race, ethnic, 
sex, and other stereotypes. 












6. Sabj.ect matter is on the level of the 
stude hts . 












7 . Information and skills presehted^^a^^ 
comparable to those used in the home, 
business, or industry. 












8. subject matter motivates stadents to learn. 












9. Subject matter is reviewed and suxhraarized. 












10. PrbgraiTi utilizes the unique capabilities 
of the micrbcbmputer to present the 
subject matter. 












II. TECHNICAL PRESENTATION ^A N/A 




1 . program is rree ol tcv-iiiiiv-ci j» ^^.^r^u-K^iw^ . 












2. The presentat ion_ rate is adequate to 
maintain interest . 












3. Information on the screen is easy to read. 












4 . Program is free of spelling and granunatical 
errors : 












5. Program instructions are easy to follow. 












6: Color increases the instructional value of 
the program. 












7. Audio increases the instructional value -of 
the program. 












3: Graphics increase the instructional value 
of the prbgraitt: 
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YES 


SQME-- 
WHAT 


NO 


N/A 


COMMENTS 


■ 

, XJ, oTUUtiNi 1. IN i Ci rvrt^ J- X WIN n i.^/ t\ 




1. Sftrtf.ents can use the program with minimal 
ass ist ance . 






1 






C*.iilAnnt^i^ Sr^t*TV7ffl\r 1 nVoXVpd 1*1 tl h e 

program . 








• 




1 c4-MHoni'c nnnhrnl thp Dace of the prociram. 












4. Students ean access the program "menu(sj" 
to change activities ■ 













«^ . Student's are permitted to change answers. 












6. Methods of responding correspond to the 
level of the program: 










— — ^r:^. 


7. Students V errprs__of entry are processed 
so that the program continues to run. 












8. Students can access available "help" and 
"hint" options at any tims. 












r. c*-i^r^ari^c r*sn <5 nt* p T* or* pxit the orociraiu 
as desired . 












10 . students control tne aequeiit-e wj- uii^ 
program. 












IV. PROGRAM INTERACTION A N/A 




1: Feedback is immediate. 












2. Cues and prompts are pruviueu ao^j-ok. 
students in answering correctly. 












3; Feedback reinforces tne correct reiat'onsaea • 












H . r eedoachw is injiiuiii.t;c»L.ciij.4i*j ■ 












5. Program helps students understand wrong 
answers . 












> T-i ^ >i r^-ii>^e ♦^Vso r^oyypr^t" cinswpi* after a * 
6 . Program gives unc t-uiict-u aiiowci. 

reasonable number of tries. 












7: Positive reinforcement is variesd. ^ 


* 










D *^ >-i-.;4*-am Kac f^Hfs abilitv to branch/loop 
depending upon students* performance. 












<=v c..^ w — ■! r» i-k r> 4- V\ <a 1 oiro 1 thp stuQent # 

9 . r SSuDaCK IS On une xevex uiic i3 1- ^ i 1 w » 












V. STUDENT EVAtOATTON V A N/A 








1. Evaluation providess a means for measuring 
attainment of objectives. 












2. Program reports which items were missed 
anca which were correct.. 
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YES 


SOME- 
WHAT 


NO 


N/A 


COMMENTS 


V. STUDENT EVALUATION — Continued 




3. Individual student, performance results 
are available to the teacher. 












4. Class performance rp'.ults are available 
to the teacher. 












program provides for printed copies of 
evalu^atibhs . 












6. Test _item__ f orraats_are suited to the 
material beinr/ tested . 












7 . Test items are clearly stated . 












8. Test item bank is provided. 












71. DOCUMENTATION A N/A 




1. Documentation is easy to understand. 












2: Docu:nentat ion is accurate. 












3. Student objectives are stated: 












4. Underlying concepts are outlined. 












fi> 5. Skills to be developed are specified. 












6 . Procedures for integrating th 

into the curriculum are provided. 












7. Fbllbw-up activities are Suggested: 

J 












8. Ppcunientat ion explains the intended use 
of support materials. 












9. Sufficient information provided to 
orerate ^he program. 












\7II; WC.^IC BEHAVIORS A M/A 




1. Program helps students identify their 
vocational skills. 












2. Program promotes pride in work. 




















3. Program promotes productivity. 












4 . Pfoqram encourages good work habits . 












. ^— / 

5. Problem solving is encouraged. 












6 . Program promotes good human relations 
s.kills. 












•7. program provides an . opportunity for work 
sacisfactibn and self -fulfillment . 












8. Program encourages creativity." 
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YES 


SOME- 
WHAT 


NO 


N/A 


COMMENTS 


VIXT: application programs a — N/A 
(to be completed for application 

prbgramis only ) - - - 





' 1. Program is adaptable to the needs of the 
stude nt . 






■1 






2, Comma hds are easily remembered. 










- 


1. Information is easily mariipulated; 












4; Corrections are easy to make. 












5. Program includes all necessary variables. 













6: Program performs reliably. 












7. Program efficiently achieves its intended 
purpose. 










V' 


3: Trial, data are_suppiied for learning to 
run the program. 












— r \ ~ 

9 . Program provides for use of printer when 
hard copy of information is advantageous : 












10. Program moves from. operat ion to 
operation isf f iciently . 












11. Program, is _cpmpatible with other 
application programs . 












12, Program has a supplementary tutorial 
program available. 













ERIC 



Parr^-C-: - Sunimary 



I. SUMMARY COMMENTS' 

identify strengths of the courseware: 



Identify weaknesses of the courseware: 



Describe uses of the program in ah instructional setting: 



2. SUMMARY OF SECTION EVALUATIONS 



Rate the quality of the courseware for each applicable section, of this form 
by checking the appropriate column; if not app.licable^ check N/A. 





YES 


SOME- 
WHAT 


NO 


N/A 


1. SUBJECT MATTER: Content has eciucational value. 
















11. TECilNiCAL PRESENTATION: Program is free of malfunctions. 










III. STUDENT INTERACTION: Students are actively involved with 
the prbqram: 
















IV. PROGRAM INTERACTION: Feedback is effectively employed. 










V. STUDENT EVALUATION: Evaluation adequately^ measures student 
pirog ress . 
















VI. ppCUMENTATldN : Rbcumeritat ibri is sufficient to ran the program. 










VII. WORK BEHAVIORS: Proqramassists students in developing 
positive work attitudes and skills. - 














VIII. APPLICATION i ROC RAMS: Program performs the task for which 
it is intendeci . 











3- FINAL RECOMMENDATION 

Check your recommendation for the courseware and explain your reasons below. 

Highly recommend R ecdtnmend with reservations 

^Recommend • D o nor recommend; . . 
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MICROCOMPUTER ebURSEWARE EVALUATION GUIDE 



NOT^: BEFORE USING THE COURSEWARE EVALUATION SYSTEM FOK THE FIRST TIME* THE 
USER SHOULD READ THIS GUIDE IN ITS ENTIRETY. 

The courseware evaluation system is intended for vocational arid teChriical 
educators, courseware developers, and persons conducting courseware reviews. 
It consists of an evaluation form and this guide. This guide is designed to 
jprbvide assistance in using the fom and clarification of each item on the 
evaluation form. The evatoation form consists of : three parts as described 
below* ' — — 

Evalaation Form Explanation 

Part A ^ 

Part A clntains descriptive information about the courseware product and 
should be filled out as accurately and completely as possible. The needed 
information may be located in the hard-copy dbcumeritatibn or within the pro- 
gram itself. Part A can serve as an initial screening device to determine 
whether the courseware review should be continued; for ex^pie, if the course- 
ware is not compatible with your hardware or instructional setting, it prob- 
ably would not be worthwhile to continue the evaluation process. 

* ' / . * 

Part B 

Related evaluation criteria arfe organized ihtc eight sections. Each sec- 
tion represents a cluster of criteria needed for courseware evaluation and 
selection. It is important to note, however, that another element must be 
added to these criteria in o^der to arriv^^at a suitable evaluation of the 
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courseware: your own judgment. Although each criterion Is ah Iraportaht indi- 
cator of quality, the overall evaluation of the courseware depends on your 
analysis of these criteria In relation to your own heeds; therefore, it Is - 
Important to keep several points In mrhd when completing this part of the 
evaluation form: 

o The criteria In the courseware evaluation form are numerous hut not 
exhaustive and represent current knowledge and perceptidns regarding 
courseware evaluation. As technology advances , interpretatidrts of what 
constltues high-quality courseware may change- 3 



o to attempt" has been made to present the criteria objectively. However, 
\Bome criteria reflect a certain degree of subjectivity and personal 

values (e.g.^ "Program promotes prdductivity") • 
'* ' * ~ . 

o No relative importahce is assigned to individual criteria, Tt& value 

attached to individual criteria is situation-dependent ; each user must 

weigh criteria In light of his dr her own situation. 

o the suggested evaluation procedure does not explicitly provide fpr 

observation'of student use of the courseware:^ This could be included In 
ah eva^luatibh^ however, at the judgment of the user. 

In completing Part B, the user should first decide which whole sections 

are applicable to the specific courseware being reviewed and then mark each 



section either — A for applicable or N/A for not applicable. Then a - 

response should be given for every criterion In each section marked A: 

o YES indicates that the criterion is fulfilled. - 

o SOMEWHAT indicates that the criterion is only partially fulfilled. 

o NO Indicates that the criterion is hot fulfilled within the program 
but should be. 

o N/A Indicates that the crlteribh is hot fulfilled and does not need to 
be. 

The COMMENTS column should be completed, at least for every itei checked 
SOMEWHAT i to explain further why that rating was given. When "Application" is 
checkedvfor Program Mode under Ihstructidhal Setting in Part A of the course- 
mie evaluakon form, Sectioh VIII should be completed along with any other 

42 



4 b 



applicable sections. This Is a separate section because of the importance of 
application pirograms In vocational and technical edacatibn and because they 
require a different set of criteria for evaluation. 

Part C 

The purpose of Part C is to provide a means of summarizing your ratings of 
the courseware being reviewed. Although a complete review using the entire 
evaluation form is recc^mended , in certain circumstances Part C could be cbm- 
blhed with Part A and used as a short evaluation or initi^il scteening device. 

Suggested Courseware Evaluation Procedure 
The following is a suggested procedure for evaluating vocational and tech- 
nical education courseware. it Is intended for beginning courseware evalu- 
ators; Experienced courseware evaiaators and professional reviewers can fol- 
low the procedure as is or adapt It In accordance with their background and 
need. The steps in the procedure are as follows: 
1- Review the evaluation guide and form.. 

2. Review the dbcumehtatibh found in the hard copy and in the program. 
This will necessitate a cursory ruh*-thrbugh of the program. 

3. Complete Part A of the courseware evaluation fdrm^ The user may not 
be able to complete all infprmatibn requested in Part A. Complete as 
many of the items as possible. 

'* , . 

4. Detemilne the feasibility of continuing the courseware evaluation. If 
there is ciampatibility between the it^s completed in Part A and user 
needs, the evaluation process should proceed. if there iS; Incompati- 
bility the evaluation process may be discontinued. 

5. Run the program as a good student making correct resjponses. 

6. Rerun the program as a poor student, making incorrect responses. 

7- Complete Part B of the courseware ;:evaluati6n form depending on the 
* program mode checked. If an application program is_ Included in the 
courseware^ complete Section VIII and any other applicable sectibris of 
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Part B. If the program mode is other than appllcatlbhi ccwiplete all 
applicable sections of Part excluding Section VIII. 

8. Summarize your ratings in Part B by completing Part C of the course- 
ware evaluation form i 

9; Decide if the courseware meets the heeds of the studentsi 

Figure t presents this si;^gested courseware Evaluation procedure 

schematically* 



44 



Begin Courie- 
ware Review 




Review Docu- 
mentation 

(Hard Copy ind 
in Projgrani) 



Complete Part A 
of the Evaluation 
Form 



Rijri^ Program 
As Good 
Student 

~~r- 



YES 




STOP 



Rerun Program 
As Poor 
Student 
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Cbmplete 
Sect I'bh VIM 
of Part B 



Complete 
Sections I -Vi I 
of Part B As 
Applicable 



Complete 
Parte 







Use 




Courseware 






STOP 



Figure 1. Suggested courseware evaiuation procedare 



AS 
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Glossary 



Branching 

Coarseware 

Disk 

Dbcumehtatlbn 

Feedback 
Hardware 

Memory 
Menu 

Progra in-JMode 
S e rie s 

Support Materials 



Program is designed so that students' pro- 
gress is determined by the specific answers 
given. 

Combination of disk (or other medium of 
transfer) and the accompanying dbcumehtatiori 
arid materials for instruction. 

Thin, usually flexible, plate on v^ich data 
or programs are stored. 

The description and instructions for use of 
a program. Documentation may be iri hard 
copy or within the program itself. 

Response of program to user iriput of 
information. 

. ^ " m _ 

Either ^ single item! or collection of 
mechanical or electronic items' required for 
use of a micrbcbmputer program. | Examples of 
hardware include mbriitbrs arid printers. 

The sectlbri of the computer where instruc- 
tibris and data are stored. 

List bf choices within a program from which 
the user makes selections. 

Microcbmputlr~uriit bf iriS traction that can 
stand alone . 



The methbd br strategy used in the presen- 
tation bf the subject matter. 

A group bf separate programs related to one 
anbther in that each program bears , in addi- 
tibri to its own title, a collective title 
applying to the group as a whole. 

items that support the act:ivitles_bf the 
persons using the program (e.g.^ studerit 
workbook). 
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^A^aJ-uatlbtl Fbriti Item Descrlptlbris 



Part A 

An explanation of each item in Part A of the evaluation form is provided 
oh the following pages. 



I. IDENTIFICATION 
Program Title 

Series Title 
Vocational Area(s) 



Subject Area(s) 



Topic(s) 

Developing Agency 

Author(s) 
Prbgramraer(s) 



Name gf the specific mlcrbcpmputer instruc- 
tional program (e.g.^ Iritrbductibh ^b 
Patterns ) • 

Name bf the micrbcbmputer instructional pro- 
gram series bf which the specific program is 
a part (eigi. Know Your Pattern is a series 
bf programs) . 

One or more vocational or technical areas 
for which the p^gram could be used: agr i-- 
culture, businessv and of f ice education, 
health, Kome ^c*^3^™ics, indas trial arts, 
marketing ^nd^^stributive education, and 
trade and industrial education. 

A more specif ic content level under the 

vo ca t to nal a r e a (e.g., textiles and " 

clothing) . 

Specific topic(s)^0-Ve3:ed in the subject 
area specified (e.g., pattern alterations). 

The organization, agency or individual pro-' 
ducing the courseware > address and phone. 

Writer(s) of the content bf the program. 

Persbri(s) writing the cbnterit in programffling 
language . 



II. HARDWARE REQUIREMENTS 

t _ 
Micrbcbmputer 

K Membry Required 



Type bf micrbcbmputer needed to run the 



prbgr ar (e.g. , 



e ire)i 



Amount of mefflory needed to run the program 
(e.g. J 48 K). ' 
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Medium of Transfer 

Prbgrammlhg Language 

DOS Specifications 
Other Specifications 

Peripherals Needed 



Means used for storing the program i^e.g.^ 
flexible disk). 

Language used to program the content (eig., 
BASIC). 

Disk operating system required (e.g.| 3*3). 

' ' . / _ . ' _ . . . _ 

Any additional requirements in relation to 
hardware. 

Any add-on hardware units required to ran 
the program (e.g., two joysticks). 



III. PROGRAM FEATURES 

Network Version 
Provided 



Program runs on a centrally located micro- 
computer arid is relayed to' numerous student 
terminals. 



Multiple Copies 
Required 



Program Can Be 
Modified 



Program Protected 



Data Disk Needed 



Disk must reanain in disk drive during, opera- 
tibn of the program;^ requiring multiple 
copies if the program is used by students 
s Imultaneously . 



Field-Test Data 
Available 



Teacher can exercise the "list" command 
(access the lines making up the program) to 
make additions, deletions, or alterations. 

Program cannot be listed (e.g."i lines making 
up the program cannot be accessed). 

Data on file disk^re required for the 
retrieval of information needed to run the 
program (e.g., empibyeSs and salaries to 
generate a' payroll) . 

The results of field testing are available 
to* prospective users of the program. 



IV. INSTRUCTIONAL SETTING 
Program Mode 



Strategy or method used to present the 
content. 

A pplicatibh ; Prbyides a service by per- 
forming a job (e.g., spreadsheet). 

prill and Practi ce; Provides repetition of 
information or skill previously acquired. 
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Educationai Gaming ; Presents facts in new 
interesting ways and provides for logical 
gueisisihg* 



Simulation ;: Presents real or; imaginary 
events, compressing extended time to develop 
prbblem--solvihg skills in a safe 
envirbxunent • 



Tutorial : Introduces new cbricept(s) and 
provides for mastery learning by giving 
immediate reinforcement • 



Student Target 
Population 

Grade tevel(s) 



Instructional Grouping 



Prerequisite Student 
Skills 

Documentation 



Student Support 
Materials 



Teacher Support 
Materials 

Correlated Materials 



Estimated Time for Use 



Type(s) of student for which the program was 
developed. 

Educational level(s) for which the program 
is intended • ; 



Instructional grouping(s5 of students v/ith 
which the program can be used* If designed 
for group use, will the program stimulate 
cooperative or competitive interaction? 

Competencies students must have before using 
the program. 

Instructions for using the program. Specify 
whether these are given in the program or in 
printed form. 

Accompanying materials for student use 
(e.g.^ handouts^ workbooks). 

Accbmpariyihg materials for teacher use 
(e.g., program guide, tests). 

The program and other instructional 
materials (e.g. , I textbook) are 
compl ementary i ; 

Approximate time required to use the entire 
program* 



AVAILABILITY 

Free, Loan, Duplication, 
Sale, Rent 



Means by which the prbgram may be_oJ?tLaihed. 



Copyright Restrictions The program ls_ copyrighted (i.e.^ reproduce 

Hon or distribution of the program Is for- 
bidden without approval). 

Back-up FoHcy Vendor makes a second copy of ^ the program 

available free or at a reducjed cost- 

Preview Policy Courserar^ Say be previewed before purchase 

under certain conditions. 

Update Policy Vendor provides revised versions of the 

program. 

Contact Name, address and phone of organization, 

agency, or Indlyldual from which the 
courseware may be obtained. 



Part B 

An explariatldn of each c^ite^ton in Part B of the evaluation form is 
provided on the following pages. 

I. SUBJECT MATTER : 

1. Subject matter is a timely practical component of the curriculum and 
reflects information or skills that can be used by students in their 
occupational fields* 

2 Learning outcomes are clearly identified for students- Objective?^ are 
presentid at the beginning or placed throughout the program to reflect 
the progressive order of the desired learning. 



3. Information is correct (e.g.. graphs, text, statistics). 

4. Subject matter is organized to reflect the usual sequence of events 
(e.g.. simple to complex^ chronological order). 

5. Racial, ethnic, or sex groups are neither overrepresented nor underrepre- 
sented! There are no inaccurate or biased generalizations about the 
characteristics of these groups. 

6. Vocabulary, readability level, difficulty of the material, and interest 
level are suited to the students. 

7. The subject matter reflects the actual knowledge and skills currently 
used in the occupational area. 

8. Subject matter is written and presented in a mariner to engage and main-^ 
tain students' interest in learning the concepts or skills. 
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9; Important Ideas and concepts are reinforced (e.g., by emphasis^ repetl- 
tlbri, questioning) i These same ideas and concepts are synthesized In 
summary. 

10. Mlcrdcdmputer capabilities (e.g., immediate feedbaclo, untiring repeti- 
tlbri) appear to provide one of the best ways of presenting the subject 
matter- 

II; TECHNICAL PRESENTATION 

Program runs consistently throughout without glitches (e.g., program does 
not stall) . 

2. The program displays text^ makes calculations, draws graphics, and gives 
feedback fast enough to maintain students' interest. 

3. Information Is displayed in a format that is well designed and unclut- 
tered. Text is not obscured by 6verlay of graphic^. 

4. Words are spelled correctly ; grammar arid punctuation are accurate 
throughout the program. 

5. tiistruccions are cbrislsterit arid uriambiguous ; complete, understandable 
directions are giveri for running the program. 

6. When color is used ^ it does riot detract from the intended purpose of the 
program (e.g., color makes material more realistic and interesting). 

7. Audio is clear in torie and understandable. The audio does not distract 
students from the educatibrial Impact of the program. The program has an 
option to delete the audio wheri desired. 

8. Graphics^ either still or ariiraated, do not detract from the subject 
matter presented. They illustrate and add meaning to the material. 



III. STUDENT INTERACTION 

V 

L. Students can use the program without excessive assistance from the 
teacher. 

2. Program promotes active rather than passive irivblvemerit of sttiderits by 
encouraging thinking and problem solving. 

3. Students have control over the ambunt of time spent bri each activity, 
thus Individualizing the iristrijQtibn tb their specific needs. 

4. A list of choices frbin which studerits can select is provided. Sksy 
access to this list is available so students can make other selections 
when desired. > 
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5. The program offers a way of going back to make changes when a wrong 
answer or response Is glvetl. -t 

6. the complexity of the type of response is based-oh the capability level 
of the students (e.g. , excessive keyboarding is not required if it has 
not been taught).; • 

7. The program is "qrash-proof It does not stop or forfeit information 
— when students either give wrong responses accidentally or try deliber- 
ately to make it fail. 

• • _^ 

8. ^ When a student requests "Help," the progr^ gives further Instructions, 

reviews previous instructions, or provides assistance in progressing 
through the program. 

9. The prograi provides students with the opportunity to exit when necessary 
(e.g., class period ends before program is completed) and to reenter at 
point ended, rather than start a^^beginriing pf the program af;a±n_- 

• • I ^ 

10. Students can chang^ the order in which they go through the program. This 
permits them to go back to review or pick up information .not covered. 

.* 

IV. J^ROGRAM INTEElACTlbN 

1. Program interacts as soon as student response is made^e.g., informs 
student of accuracy of answers^ preseiits further information, or explains 
previous information)* 

2. if the wrong answer is gi^eri. the program provides further'^ information or 
^lies (e.g., number of letters in the correct word is provided). 

3. ^rrect responses are recognized in a posiTlv^ manner ( e.g . i student is 
complimented on corretit answer). The program* s response to incorrect 
answers is not so interesting that incorrect responses are encouraged. 

^ -f ..*____ 

4. Students are not addressed in a derogatory manner (e.g., "You dummy") 
.when incorrect answer is given. 

5. - Progr^ does more than merely review the material; it provides the reason 

"that the answer is incorrect (e.g., "Answer B is wrong because . . * )\ 

6. The student is not permitted to continue making incorrect answers indefi- 
nitely. It is riot possible to arrive at the correct answer by the pro- 
cess of elimiriation; * 

7. The type of positive reinforcement changes as the program progresses, 
since feedback such as "You* re terrific" becomes tiresome when overused. 



54 



56 



8; Program offers activities based upon the student' s_ responses. Branching 
offers alternative activities, with different levels of difficulty or 
interest. Looping is a repeat of the activity for review. 

9. The type and content of the feedback are geared to student cbmprehenslon . 



V. STUDENT EVALUATION 



Evaluation Included In the program (whether test items or performance 
type) Is based on the stated student objectives and indicates prbgres^s 
toward attainment of the objectives. 

The results of each student's performance on the evaluation are provided 
by the program (e.g., test score, items correct^ Items wrong). This 
Information Is protected by a separate password for use by the Individual 
student and the teacher. 

Program Identifies for individual students the Items for which correct 
and Incorrect responses were made, this assists students and teachers In 
understanding what corrective measures need to be taken. This Informa- 
tion is protected by a separate password for use by the individual stu- 
dent and the teacher. 



A composite view of class perfbrmarice is given ( e .g . , average, range, 
percentiles) on the evaluation. This irifbrmatibn is protected by a 
separate password for use by the teacher. 

Hard copy of both individual student and cbmpbsite class results is 
available to the teacher to facilitate record keepirj. Hard copy of 
individual test results is available tb the student. 



The type of test Item used (e.g. 3 true-false, multiple choice, perfor- 
mance) is varied to reflect the best method of determining student 
attainment of objectives. 



Test items are easy tb understand. Content and vocabulary are consistent 
with those in the subject matter presented. 

A data bank bf test items* provides the teacher with ^he capability bf 
generating tests by a random sampling of Items. 



VI*. DOCUMENTATION 

1. ThC\anguage, vocabufery, and organization of the material in the docu- 
'meritatibn are easily comprehended. ^ 

2. All informatiott is correct (e.g. , graphs, text^ statistics). 
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3. Expected learning outcomes are listed. i£ particular skills are to be 
developed, they are specified. 

4. An explanation :of the ideas and principles from which the program Was 
developed is given. ; 

5. The particular skills to be learned through using the program are stated. 

6. The teacher is given specific suggestions on whe^e and how to combine the 
program with the existing curriculum.. 

7. Suggested follow-up activities geared to the students are given to rein- 
force the information presented. 

8. Recommendations on where arid how to use all student materials are given. 

9. All necessary informatibri is provided so that teachers or studerits cari 
run the program from start to finish regardless of prior experience. 

VII. WORK BEHAVIORS 

1. Students are made aware of their competencies in relation to their 
Intended occupations. Iridividual strengths and weaknesses can be deter- 
mined and used as guidelines for further development. 

2. Program presents all Work as tasks to be approached arid carried out in a 
conscientious manner. Regardless of the nature of the Work, students are 
always encouraged to "give it their best effort. 

3. Prbgram encourages the achievement of maximum outcomes through the use of 
available resources. 

4 Positive behaviors are advocated for getting, performing, and keeping a 
job. These behaviors include depejadabilitys purictiiality, cooperation, 
arid iriitiative. 

5. Program encourages students to solve problems and make decisions that 
have transferability to their occupations and everyday lives. 

6. Emphasis is placed on "people skills"~the ability to communicate and get 
along with people. 

7. Students complete the prbgram feeling that they have accomplished^sbme- 
thing. Equally important^ the means. Of accomplishment leaves students 
feeling good about themselves arid their ability to complete the task. 

8. Iridividual creativity is promoted through the opportunity to develop hew 
ideas, products, or ways of performing tasks. 
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Vlft. APPtiCATIbN PROGRAMS 



1. Program offers sufficient versatility and detail; that the coverage and 
complexity of the program can be changed to meet the specific needs of 
the students using it. 



2. Specific commands or instructions to enter and niarilpulate data are 
logical in nature and simple to use. 

3. Process required to change data (frequently numbers) is simple to under- 
stand and easy to use. 

4. Information being used in the program can be corrected or changed at any 
time without having to rerun the entire program. 

5. All fields and variables necessary to perform the task are available, or 
the program is adaptable so the necessary variables and fields can be 
added • 

6. Program provides the same answer or outcome each time, so that one can 
depend on its accuracy. 

7. Program performs the task it is supposed to do. 

8. Supplementary information or data source is provided to use in learning 
to run the program. 

9. Program provides for printer use when hard copy of the resulting informa- 
tion is advantageous. 

10. The sequence in moving from one operation to another is easy to under-, 
stand and implement • 



11. Program is either bundled (designed to be. compattblQ with other applica- 
tion prdgrrtns) or integrated (developed specifically to be combined with 
other particular application programs). 

12. Tutorial program presents the concepts' and information heeded in learning 
to operate the application program. 

Part C ^ 

An _ explanation of each item in Part C of the evaluation form is provided 
as follows. 



1. SUMMARY COMMENTS Desc^be the advantages ^of this particular 

courseware. ^^_P??sibie, compare it with 
other courseware reviewed. 
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Describe the disadvantages of this 

particular, courseware- If possible, compare 
it with other courseware reviewed. 



2; SUMMARY OF SECTION 
EVAttJATiONS 

3. FINAt REGCWMENDATION 



Expiaih different ways the courseware might 
be used in learning situations, both in the 
classroom and in informal settings. 

Summarize the rating of the courseware by 
section of Part B of the evaluation form. 

Give a final overall rating of the course- 
ware for instructional use and a brief 
explanation of why that rating was given. 
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CHAPTER iV 



REeOM^ffiNDATiONS 



The major recommeridation emerging from this study is that a centralized 
agency or network of agencies provide national leadership in microcomputer 
courseware availability and evaluation for vocational and technical education. 



The National Center, or the National Center In cooperation vd.th the National 
Network for Curriculum Coordination in Vocational and Technical Education 
(NNCCVTE) , could provide the leadership needed* The Vbcatibrial Education Cur- 
riculum Materials (VECM) database^ developed and malhtairied by the National 
Center and NNCCVTE, couTd^^'^rve as the vehicle to disseminate irifbrmatlon on 
the avilablllty arid evaluation of micrbcbmputer courseware* 

Based on the activities of this study arid the many requests (over 50 in 
the last 3 mbriths) received from the field, the 2 major areas of concern 
related to micrbcbmputer courseware to be addressed are evaluation and 
availability. 

The following activities are suggested to provide national leadership in 

implemeritirig A System for Evaluating Microcomputer Courseware for Vocational 

and Technical Ediicatibri arid in providing information: on the availability of 

micrbcbmputer courseware: 

d &isBem4nate Information about the courseware evaluation system . A 
dlssemiriatlbri plan should be developed and implemented, that will make 
the cbursewate evaluation system w^ely available, information about 

system can be made available through existing National Center tnech- 

anisms such as electronic newsletters, printed publications ^ confer- 
ences, and workshops. Existing vocational and technical networks and 
agencies such as NNCCVTE and the National Postsecondary Alliance could 




disseminate Information about the syston. The courseware evaluation 
systei can be made avilable through NNCVTE, ERIC, and the product cost- 
recovery system at the National Cetiteri 

Solicit endorsement of the course ware e v al uat ton system fr o m vocational 

.rnfpfiRlonal o^i^antza tlons (e.g., American Voc^- 



^tnd^ technical education p 
tional Association^ American Home Eco 



^^octation, and American 



Association of teacher Educators 



:^are)< 



Endorsement by these 
organZzatlons'would prrabte the use of the system throughout the field 



on a national basis. This in turn 8hoa3.d lead to the develbpment of 
better-quality courseware for use in vocational and technical 
education . 

-n±±abor ate with publishing cbmpaa4eg^nd^ather agencies to faclllta^e 
the adoption -a nd use of tha courseware^ ^vaiuatlon system . Interest^has 
been expressed by McGraw-Hill, the National Education Association (NLA), 
and the Northwest Regional Education Laboratory (NWREL) to engage In 
collaborative efforts in courseware evaluation with th,e National Center. 



Conduct t echnical ^slst a nc e programs on t he courseware evalua£±o g_lZ8- 
tem. Although the evaluation system can be used by individuals without 
specific training, it would be more beneficial, ^If technical assistance, 
including hands-on experience, were provided. Expfeclally developed 
training materials would coSplement the technical assistance programs. 



o Provide leadership 



uduct and dlssemihatibn. of 



and 



technical educ ation c ourseware reviews . Leadership should be given to 
implementing professional courseware reviews using the new evaluation 
system. It is suggested that at least three vocational and technical 
educators trained by the National Center evaluate a given courseware 
product using the evaluation form. Their evaluations would be syn- 
thesized before a review Is published. 



o Dlssemihat e pfof e s s ln nai coursewar e reviews thr 
Isms such as Nattonai Center publlcatibhs aitd « 



sL biished mechah- 
newsletters* 



Tnfbrmafcio n^^ut cour seware reviews could ^b^^aade availab le throusV 
VEC>n — Professional courseware reviews are a valuable screenxng device^ 
Ihit practitioners can use in selecting courseware they wish to evaluate 
further. - 

o Field-te st the cour^ewar^ evalu a t i on system . Extensive field testing of 
the courseware evaluation system should be conducted to further refine 
the system and 'to implement a first update when needed. 

tem annually to determine if-Xhere 
Revise when necessary. Field-test whenever 



o Review the courseware evar! 



±s-^ need for updating , 
revisions are extensive. 



ExpTor^ the-possibilit:y of expanding the^ VECM dat abase 



to Inc lude InfoT-- 



ma 't^l^fil^^'c^^^e^^Iaii;" produ ced c^ mi rs ewa r B as well ^» P^^^^ ^-^ggS 
courseware. Since one of the greatest concerns in tfte field^currently 
is locating courseware, the National Center and NNCCVTE could serve 
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practitioners well by Including commercial courseware products in VECM.. 
This V70uld provide not only a much needed service bat also eliminate the 
current restrictibri bri entering a" public doaiain product that might 
become available commercially. 

Expand the scbpe bf the eval uat i on s y a^em to include th e means for 
collecting student ratings cour s e ware for teacher use . Students 
could be questioned on the pace of instruction, the amount of material 
covered! the "interest factor", and the difficulty of the courseware, 
this informatibn cbuld then be used to validate the teacher's evaluation 
of courseware. 
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A. technical Panel 1 



Particip ants 

Glinda B. Crawford, Ph.D. , Assistant Professor 
Departirierit of Home Ecbrioinics arid Nutritiori 
College for Human Resources 
University of North Dakota 
Grand Forks ^_ND 58202 
(701) 777-2539 

J. Richard Dennis, Ph.D., Associate Professor 

Def5ar tment of Secondary Education 

Cbmp u t e r- B as ed~E du ca t i on 

College of Education 

University of Illinois . 

395 _ Education Building 

1310 South Sixth Street 

Champaign, IL 61820 

(217) 333-4664 

Molly Hepler Langs taffj Editor 
CONDUIT 

University of Iowa 
M310 Oakdale Hall 

— P-:err~Bux-C 

Oakdale, _IA__52319 
(319) 353-5789 



Judith Lamb recht, Ph.D., Associate Professor 

Division of Business and Marketing Education 

Department of Vocational and Technical Education 

University of Minnesota 

420 Vocational Education Building 

1954 Bnford Avenue 

St. Paul, m 55168 

(612) 373-0112/3330 

Judith Roderisteiri, Ph-D., Project Associate 

Vocational Studies Center 

School of Education 

University of Wisconsin-Madison 

1025 West Johnson Street, Room 964 

Madison, WI 53706 

.(608) 263-4367 
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nfirraarion Letter . to Pahelists 




th« Ohid Siai* Unlvmiiy 




L M iATIIiALaMTII: :_ _ 

r fO| KSUUCH M VOCATOUL EOOC*nOI1 



l960_Kenny Road 

Columbui.Ohio 43210 



Phooi; 6^4-486-3655 

Cable: CTVOCEDOSU/Columbus. Ohio 



August A, 1983 



Dear 



We are pleased that you Have agteed to serve on the panel to assise us 
in developiug a system for evaluating microcomputer instructional software 
for vocational and technical education. 



The panel will convene at the National Center^ September 21-22, 19B3. 
Arrangements for lodging have been tnsde for you at the Hilton Inn-University. 
3110 pientangy River Road for the nights of September 20 and 21. _ You will 
need to take a taxi frgm_the airport to the Hilton Ihh. You w^^^I^^ ®^^,^" _ 
Che Hilton Inn_ lobby on both September 21 and_22 at S:OD a.m. to be transported 
CO che National Center for the meeting to start at 8:30. 

Enclosed are the follbwlhg materials: 

1. Agreement for Services (CO bi signed a^ returned tnshediately) 

2: Travel Guidelines and Consultant Expense Report 

3. Project Prbclle 

4. Tentative Agenda 

Please sead us your flight schedule (arrival and returning times) and a . 
vita in addition to Che signed "Agreement for Services". form_ as soon as possi- 
ble. _Feel free to call, (800) 848-4815 , If we can be of assistance. If I 
am out, please leave a message with Trudi Richardson, secretary. 



We aire looking forward to working with you. 



Sincerely , 



Shirley A. _Cha3e, Ph.D. 
Project Director 



5AC: Cgr 



Enclosures 
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Agend a 



Wedn esday. Au^ us^li^ 



8:00 Pick-Up at Riltbh Ihh 
8:30 WeicoSe and Ihtroductibhs 
8:45 Project Orientation 

Workshop Dbj ectives _and Procedureis 
9:30 Review and Rating of Evaluation Fdirin 



Break 

10: 1:5 Development of Evaluation FbrTri 
Lunch — AccuRay Cafeteria 
1:00 Development of Evaluation Fortil 

Break 

. ..... .... tjf 

3: 15 Develbpmeht of Evaluation Form 
Seiectibh of Response Format 

Thursday, August 25 



00 Pick-Up' at Hiltc^n Inn 

30 Develbpmeht of Evaluation Form 



Break 

10:00 Testing of Evaluation Form 
Lunch — AccuRay Cafeteria 

12:30 Final Recommendations on Evaluation Fbrm 
Suggestions for Guide 
Closure 
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by Panel II Partlci^fiafrt^ 



Title 



"S I* 



Drivers' Educ^ 



[ndu St r iai Arts 




Know 



MECC 



Lane Cotmanity College 



Orange Juice 

Software Systems 



Reprodu ction 



Micro Power and 
tight Company 



The Daily Mena Analyzer 



Orange Juice 

Software Systems 



Autdmdtive Technician Math 



MECC 
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B, Technical Panel II 



Participants 
Jim Fay, Ph,D. 

Freshwater Informstion Services 
624 Hackberry Street 
Box 1^1 

Chatham, iL 62629 
(217) 483-5772 

Helen B. Hbvls^ Assistant Professor 

Heme Economics Education 

113 Ackerman Hall 

College of Home Economics 

Indiana University of Pennsylvania 

Indiana, PA 15705 

(412) 357-2563 

Wanda Monthey^ Program Arvdlyst - 
Vocational Education Division 
Oregon Department of Education 
700 Pringle Parkway, SE 
Salem, OR _9?310 
t503) 378^2713 

Harley Schlichbing, Ph.D. , Director 

Ins; true tional Mat j rials Laboratory 

Uniyersicy ' .r Missouri 

10_ Industri-^i Education Building 

Columbia, MO 65211 

(31''i) 882-2G83 



'une Schmidt, Ed.D., Assistant Pro.f^^ssor 
Vo. ridnal Technical Education 
}5uh . ..iess Edtication 

Virginia Polytechnic In: titute and State Univ-rsity 
213 Lane Hall 
Klackt^burg, VA 24060 
(703) 961^ 5471 



69 



>9 



ERIC 



Cbnf irmaribn Letter -t- b Panelisrs - 



Th« Ohio SUt« Untv«r»(ty 



mmimkavm 

FOt-Res£»CM m vocinoiuc coocxdon 



1960 K«nny-Ro«d 

Columbul. Ohio 43210 

Phone: 614^^6-3655 

Cable: CTVOCEDOSO/Columbus. Ohio 



August 10, 1983 



Dear 



_Ve are pleased chat you have agreed Co serve on c he panel Co ass is E us 
in developing a syscem. for evaluating microoompucer inscruccional sofcware 
for vocational and cechtiical educacion. j 



The panel will convene ac Che National Center, SugusC 24-25, 1983. Arrange- 
mencs for lodging have been madeforyou ac the Hilton Ihh-UhiversiEy . 3ttO 
dlentangv River Road for the. nighcs of i^gusc 23_ and 24. You will need to calce 
a caxi from Che airpbrc to Che Hil£on_Inn. You will be raec in_ch6_Hilcon Inn 
lobby on boch_Augusc 24 and 25 ac 8:00 a.m.^co be transporced co che JJacional 
Center for che meecing co scare ac 8:30. 



Enclosed ate che following macerlals: 



1. Agreemenc for Services (cb be signed and returned immediately) 

2. Travel Guidelines and Consultant Expense Report 
3: Project Profile 

4. Tentative Agenda 

Please sendus your 'flight schedule (arrival and returning times) and a 
vtca in addition to the signed I'Agreement for Services" form as soon as 
possible. Feel free to call, (80b)_348-4815, if we can be of assistance 
I am out, please leave a message with Trudl Rrichardson, secretary. 

We are looking forward to working with you. 

Sincerely , 



Shirley a: Chase, Ph:n. 
I-ro-ject director 



Tf 



SAC.-cgr 
Enclosures 
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Informaribh Letter to Panelists 



Avyuac 29, 1983 



• Dear 



Kenny- Road 

Columbus, Ohio 43^10 

Phone: CI-' ^^4^^36BB 

C*5b: CTVOCnDOSO/Cotiimbus. Ohlb 



The project: to develop a syistita tor evaluating al*i:rbcdm?uLer iriiiruetidhal 
software for vocacionai educacior. is weJLi under way and we ar^ luuklng forward 
to working with you on Sepcenber 21-22^ 

To bring you up Co date on our acCJlvlCles, we. are enclosing _a bibliography 
of Motcware evaluation literature and d list of the evaluation forms we Have 
acquired and examined. The preliminary form developed by staff after reviewing 
these materials is npw_ being revised to incorporate suggestions made by the 
firsc panel which met August 24-25. 

We hope that you will help u» finalize ah evaluatibr; fbtm ;»i'ccific to the 
needsof vocational and technical educators. We envision^ that this form, 

accompanied by a guide'fpr its use, will be used by_yocational and techni 

Ceachers tn selecting software, reviewers of software, and software developers. 

The agenda for the two-day session will include: 

1. Review and recommendations for the evaluation form. 

2. ^ Testing of Che finalized form. 

3. beveiopmeht of a user guide. 

A packet of material, includ.i.ng__the revised evaluation form, will be waiting 
For ybu on your arrival at the Hilton. 

We welcome your ideas and assistance oh this project. If ybu have any 
evaluation forms not on the enclosed list, or references to other pertinent 
ttcerature, let us knov?" or bring them with you. Also, we are in need of 
additibhal vocational and eechntcat edacation software to test the form. 

Contact us (800) 848-4815 if ybu have any questibhs. When I am hbt In 
Lhe office, leave a message with my :secretary, Trudi Richardson. 

if ybu have hbt sighed and returned ybiir "agreement for Services" form, 
please do so along with a vita for our records. 

Sincerely i 



Shirley A._Chase, Ph.D. 
Project Director 
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iber 21 

8:00 Pick-Up at Hilton Inn. 
8:30 Welcome and Iritrbductions 
8:45 Project Oriehtatibn 

Workshop Objectives and Procedures 
9:00 Explanation of Form recommended by Panel I 

Review of Coursewaria Using^- Evaluation Form„ 
9:45 General Comments/Discussion on Evaluation Form 

Break 

10:15 Refinement of Evaluation Form 
Lunch --Ac cuRay Dining Room 
1:00 Further Development of Evalua,tion Form ^ 

Break 

3:15 Further Development of Evaiaatibn Form 
eiosure 

Thursday. September^^ 
8: 00' Pllck-Up at Hilton Inn 

8:30 Opening Remarks _ __ _ ^ ^ 

Review of Courseware Using Panel II Evaluation Form 

^ ' Break 



10^15 Final Revision of Evaluation Form ______ 

Individual Recommendations for Implementatxon of Form 

Development of Guide 
Lunch — AccuRay Dining Room 
1:00 Development of 'Guide , 

Break 

3 • 00 Development of Guide . _ . 

Summary— Written Recommendations for Evaluation System 
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Cgu r s ega r e- -Re v t e wed and Ev a 1 ua t ed 
W Panel II Parttctpants 



Caf : 



ne 



'^L • ygrs* Edacation/Industrial Arts 
(Microineter) 

Know Yoar Pattern 



Antomotive Tecni* xc '/u^ Ma^t; 

Heatioss 

Conipete 

T he Clinical Dietitian in 
Nutritional Care Process 

The Ruler--Quarters 



Engine 



CPS: 



Ca re e ^-P Ian r 



Source 

Lane Coiranunity College 
MECC 

Orange Juice 

Software Systems 

MECC 

MECC 

CONDUIT 

OSU College of Medicine 

Southeast Oakland 

Vocational-Techhical 
Education Center 

Southeast Oakland 

Vocational-Technical 
Education Center 

The Coriover Company 



wlt4v Mi^^ 
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•e. Pilot Test 



Charlbtte Adamaszek 
Home Economics Teacher 
Brbbkhaven High School 
4077 Karl Road 
Columbus, OH 43224 

Kent S. Anslxnger 
Biiisiness Data Frocessing 

Tab Supervisor 
Columbus Technical Institute 
j50 E. Spring Street 
CcLuinhus, OH 43215 

Jeri Bradford 
Hcrticulture Instructor 
Northeast Care^»r Center 
3871 St^lzer Road 
eoiumbus, OH 43219 : 

Karen L. Burger 
Nursing Instructor 
Columbus Technical Institute 
556 E. Spring Street 
Columbus, OH 43215 

Thomas Clyde Elliott 
Vocational Agriculture Teacher 
Jonathon Alder High School 
Plain City, OH 43054 

Patricia Fein 

Nurse Assistant Instructor, , 
Fort Hayes Career Ceriter 
545 Jack Gibbs BouliavatiJ 
Columbus^ OH 43215 

Candice Hill 
COE Teacher 

Ceriteririiarl Higii. School 
1441 Bethel Road" 
Columbus, OH 43220 - 



Kenneth Kibbey 
Teacher Coordinator 
Distribative Education 
Whetstone High School 
4405 Scenic Drive 
(Soiumbus, OH 43214 

-Richard Weese 

-^JT & I Drafting Instructor 
Northeast Career Center 
3871 Stelzer Road 
Columbus^ OH 43219 
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Coarseware Reviewed and Ev aluated 
by teacher Reviewers 

Title 

Caffeine , - 

Hort-Plaat- 

■ Khbv Your Pat t:ern 

Manure Fe^to^^er Vaiae 

Micrb cdmpute r Keyboarding 

Drivers ' Educat ton/ Indust rial 
Arts (Kicrometer) 



7^ 



So urce 

Northwest Vocational 

Curriculum eoordinatidh 
Center 

Northeast Curriculum 
Coordination Center 

Orange Juice Softv7are 
Systems 

Instruct iohal Materials 

Services 
Department of Agricultural 

and Extension Education 
The Pennsylvania State 

University 

South-Westerh Publishing 
Company 



SECC 
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Review Fdrin for Evaluatj 



I. Evaluatibh Foriii 

it 

A. What were your general reactions to the form? 

1. Content: 

S 

2, Format: 



B; Explain any difficulties you had in using the fortti. 
e. What suggestions do you have for improving th*2 form? 

m 

\ 

i). How do you see this form being used in vocational edacationi 



ii. Guide 

A, What were your general reactions to the content of the guide? 



B. Explain any difficulties you had in using the guide. 

C. What suggestions do you have for improving the guide? 



III. What suggestions do you have for the procedure (steps) to follow In 

reviewing ano selecting software using the evaluation form and guide? 
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D, Supplemental Revj 
Disseihihatioh and Utiliza tion Cdriferen oe 



Reviewers 



Leota Boesen 

Assistant Information Specialist 
Vocational Education Services 
840 State Road 

4& Bypass,; Room 111 

Bloomington, IN 47405 

Jo Brooks, Information Specialist 
Vocational Education Services 
840 State Road 
46 Bypass^ Room 111 
Bloomington, IN 47405 

Steve Franks^ Extension Associate 
Vocational & Adult Education 
Auburn University 
203 Petrie Hall 
Auburn, AL 36849 

Jerry Grbyer, Specialist 
Industrial Arts 

Utah StateOf fice of Education 

250 East 500 'South 

Salt Lake City, UT 84111 

Bill Jacobseh, Assistant Professor 
Occupational, Adult, S Safety 

Education Diepartinerit 
Marshall Uriiviersity 
Huntington, WVi 25701 

Carol K. Latighlin, Director 
Vocational Currlculuir Resource 

CiBritiBr 
750 Mar ret t Road 
Lexington, MA 02173 

Barbara A, Luckner-Loveless 
Associate Director 
Western eCC 
Han© lulu. Hi 96822- 



Jbhri MacKertzie, Assistant Coordinator 

Vocational Education Resource Center 

Michigan State University 

133 Ericksbn K rll 

East Lansing, MI 48824 

Beatrice Melendrez, Coordinator 
Vocational Information & Program 

Services 
University of New Mexico 
351 Rio Community Boulevard 
Belen, NM 87002 



Jan"" Novak, Project Director 
Wisconsin Vocational Studies Center 
256 Educational Sciences Building 
1025 West Johnson Street 
Madison, WI 53706 

Leslie Peters, Resource Specialist 
Pennsylvania Department of Education 
333 Market Street, P/0. Box 911 
Harrisburg, PA 17108 

Martha Pocsi, Director 
Northeast CCC 
Rutgers University 
200 Old Matawan Road_ 
Old Bridge, NJ 08857 

Barbara Reed, Supervisdr-iri-Charge 
Vocational Home Economics 
Ohio Departinent of Education 
65 South Front Street 
Columbus i OH 43215 

Vila Rosehfeld 

Professor arid Chairperson 

Home Ecoribmics Education 

East Carbliria University 

Greeriville, NC 27834 / 
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John Roth 

Vdcatibhal Educatl oh Specialist 
Departtneht of Education 
State Office Building 
Mbritgdiiiery , AL 36130 

Reitia A. Roybal, Coordinator^^^ 

Vocational Information & Program Services 

University of New Mexico \ 

351 Rio Coininunity Boaievard 
Helen, NM 87002 

Antionette W. Weich^ "^^i^ 
Vocational Education Consultant 
Agricultural Education eurficuluiri , 
Materials __ _ ^ 
The Ohio State University v 
2120 Fyffe Road 
Columbus, tm 43210 
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* 

:btovember 19B3, 3:10 p. 

Kenny Road, Room B 

1. Orieritatibri 

A. Welcome and iritrdductions ^ / 

B- Project background /' ' 

C. Purpose of session 



1. Providia participants experience in evaluating courseware 

2. Obtain isuggestions for improving evaiuatioh system, 
II. Courseware riaview using evaluation form 

:il. Discussion session on suggestions for improving evaluation form and guide 
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Courseware Revia^ 



liuated by 



Disseiiiiriat iotv ^- 



■Conference Participants 



Title 



Autdt 



C affeine 



Compete 



Data- Entry Activities for 
the Microcomppter 



Developmentai Stages 

i::ht:xvBr s-^^^^ ion/ Indust r ial 

Arts (Micrometer) "~ 



Grab-a-Byte 

Heatxoss 

Hort^Piant 



Know Your Pattern 
Microcomppter Keyb oar ding 
Agriculture Application P^o^^ms, 

P arts of Speech 

E nterprise Sandw ich Shop ^: 
" a Market Simula^i^^t 

The Micro-Editor 

The Sentence 



Source 
MECC 

Northwest Vocational Curriculam 
Cbbrdihation Center 

CONDUIT 

South-Western Publishing Company 

Northwest Vocational Curriculum 
Coordination Center / 



MECC 

Ohio Dairy eoancil 
MECC * 

Massachusetts Vocational vlurriculum 
Resource Center : 

Orange Juice Software Systems 

South-Western Publishing Cbiripahy 

InstTfactional Materials Service 
Department of Agricultural and 

Extension Education 
The Pennsylvania State University 

In'st rue t iona 1 Mat e r ial s Labora to ry 
University of Missouri-Columbia 

McGraw-Hill 

South-Westerh P^ Company 

Instructional Materials Laboratory 
University of Missouri-Columbia 
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the commercial and public domain courseware products reviewed daring this 

study are listed in the fbllbwihg pages* The vocational or technical area, 

courseware titles and availability_irif ormation are piesented- Each courseware 
product vas reviewed on an Apple lie microcomputer unless otherwise specified. 



Area 



Agriculture 



Title 



Availability 



Compete 



CONDUIT 

P.O. Box 388 

Iowa City, iA 52244 

(319) 353-5789 



*'D_Feed'' Dairy Rationing 
Balance Trdgratn 



District One Technical 
institate-Eau Claire 
62G W. eiairemont Avenue 
Eau Cialre, WI 54701 
(715) 836-3900 



Bu^l^ ing - 



ru c t i o n and 



Ele G t r4c-^lQ t 
Review 



Lew 



ind Related 



Hobar Publications 
1234 Tiller Lane 
St* Paul, MM. .55112 
(612) 633-3U_0:- 



id Related Review 



S .-Q ,:E^.jP^/M4scgllaneoas 



Instructional Materia]. s Service 
Department of Agricultural and 

Extension Education 
The Pennsylvania State Universit; 
13 Armsby Building 
University Park^ PA 16802 
(814) 863^3824 



Hort-Piant 



Massachusetts Vocational 

Curriculum Resource Center 
758 Marrett Road 
Lexington, MA 02173-7398 
(617) 863-1863 



Farm Accounting and 
Records Management 



Specialized Data Systems 

Incorporated 
P.O. Box 8278 
Madison, WI 537138 
(608) 241-5050 



SI 



title 



Availal>il-i^ 



Business 
atid Office 



Magic Words — Spell Cteckxitr 
Magic Words IT 



Art Sci . _ _ 

1G432 Burbahk Boulevard 
North Hollywood^ CA 91601 
(213) 985-2922 



Health 



Home Economics 



Typing Tutor (- TRS-8Q ) 
Typing Tutor II (TRS-8 0) 
Budgeting Tutor-4al^ 
Budgeting Simulati-on^ 

Microcomputer Keybbarc3444g^ 
Data Entry Activities^o^^ 

the Microcomputer 
The Micro-Editor 



EMC Publishing 

300 Ybrk Avenue 

St. Paul, m_ 55101 ; 
(800) 328^1452 

South-Western Publishing Company 

5101 Madison Road 

Cincinnati, OH 45227 

(513) 271-8811 or (800) 543-1985 



Cai:- 




Nbrthwest Vocational Curriculum 

Cpordination Center 
Building 17 Airdustriai Park 
Olympia, WA 98504 
(206) 753-0879 T5V0 



ikptatiort Syndrome 



Shoc k 
Stressor 

Copi ng With Stress 
Nursing'^Math — Conversio ns 
Introdu ction 

geproduct ion 



Micro, Power and tight Co, 
12820 Hfir^^et^ei^^a-d- 
Suite 224 
Dallas, TX 75230 
(214) 239-6620 



Jumping Jac k Flas h-!_ 
Grease __ 
Food For Thou^h^ 
You Are What- You Ea t 



Dietary Data~Anarysis (DBA) 
P.O. Box 26 > 
Hamburg, NJ 07419 
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Title 



Aval labi -l-i L-v 



Home r. CO no mil 



i.-n-^:0e a 1 1 on a j^ ^d u c a 1 1 o ik 
Home- 4jIco Mbmlc .s-^ 
Home Ecbribmics II 



Illiridis Vocational Curriculum 
Center 

Sarigamdn State University 
Springfield, IL 62708 
(217) 786-6*375 



EKLC 



One-Baking 



Masisachuse tts Vocational 

Curriculum Resource Center 
, 58 Marrett Road 
Lexington, MA 02173-7398 
(617) 863-1863 



Pd isdn Pr ao^-Ymir Home 
Income Meets Expenses 
You Can Bank On It 
Comparative Buying 
Analyzing An Ad 

Nutrition 

Health Maintenance 
Food Facts 



MCE, Inc. 

157 South Kalamazoo Mali 
Kalamazoo, Mi 49007 
(616) 345-8681 



Mxnn<=.sota Educational Computing 

Consortium (MEGC) 
2520 Broadway Drive 
St. Paul, m 55113-5199 
(612) 638-0683 



Grab-A- Byte 



Ohio Dairy Council 
-2929 Kenny Road, Suite 190 
Columbus, OH 43221 
(614) 451-2864 



Place Setting, Mea l 

Service ^ and Table Manners 
Know Your Pat tern 
dSP a nd The Principles of 

Egg Cookery y Quickb rea d 

Preparation and Vegetable 

Cookery 
Food Poisoning^ Sanitation 

Preservation 
Daily Menu Analyzer 
Design Elements and P-^ ln<: ) p-i^s 
Nutrition ^an^ ^^--4-^a^^4^ 

j^ od G r^mpnS 
To Salt. ■ .O r ^Jn^- Tn Sn1r?-~ 

TliG Sodiur., /uui. 



Orange Juice Software Systems 
222 South Washington Avenue 
New Richmond, _WI 54017 . 
(715) 246-3588 



Area 



title 



— - — ^v^i-la bi l lty 



Indus trial 
Arts 



>larketing and 
Dis tr i bucive 
Ed uc a L i on 



Industrial Art^—Bas^ 
C omputer 'Pro- ams 



Microcompute r Appl ^c^fcion^ 
fo r tfe-^art a Pro Ges^slitg 
WorR;_K.i^ . ■ 

E r v r i s e S a n dx^ i -ch Sh o p s : 
^ larket ^ Imilatidu 



Iowa Department of Public 
Instruction 

Grimes State Office Building 
Des Moines, TA" 50319 
(515) 281-4711 



McGraw-Hill Book Company 
Book Distribution Center 
Princeton Road 
Hightstown, NJ 68520 
(609) 426^5006 



4rnvaivt d ry 



;rndc and Microcomputer Applications 

Industrial in Vocational Education: 

Trades and Industry t 
Trades and Industry II 



MIND,- Inc. 

50 Washington Street, Suite 12{ 

Norwalk, Ct 06854 

(203) 846-3435 or (800) 243-58: 



Illinois Vocational Curriculum 
Center 

SangambnvState- University 
Springfieid,_IL 62708 
^(217) 786-6375 



Mechanics — Engine Tune Up 



MIND, Inc. 

50 Washington Street, Suite 12( 

Norwalk, CT 06854 

(203) 846-3435 or (800) 243-38: 



Heatloss 



Auto Mat:h _ _ 
Drivers ' Edu- eatibri/ 

In^u&t ^ial Educa fclor ^ 

(Mi^^ 

iia4^ S tyJ-ing — Fac i aj' 
Shapes 



Minnesota Educational Compotinj 

Consortium (MECG) 
2520 Broadw^^ rDrive 
St. Paul, MN 55113-5199 
(612) 638-0685 

: Oswego County Board of 
Cooperative Educational 
Services (BOCES) 
P.0. Box 488, Route ?64 
Mexico, NY 13114 
(315) 963-7251 



Aval lability 



Vbcat iiMial 
Edu cat Ibii , 
Gene ral 



Afe€^t^i-o nal Math 



Northeast Network 

Cubrdinatic^i 
Rutgers Unlveriii'i: 
200 Old Mat aw ar. 
Old Bridge, NJ 
CiOl) 390-1191 



for Curriculum 



Kcad 
08857 



Parts o f Speech 
The Sentence 



Insrrtjrtional Materials L<ibdratdry 
Universir;- Missouri-Co luttibia 
10 Industrial Education Building 
Columbia, MO 65211 
(314) 882-2883 



Vocat Lonal 
Guidance 



Apple Pilot Bemo Disk 
Freshmen Explore 



Ma: .sachase t ts Vocat iona 1 

Gnrricuiom Resource Centc^r 
758 Marrett Road 
texington, MA 02173-7398 
(617) 863-1863 



Career Planninj^ The Cor. over Compani 

S>..tem w i th Microcom puter s P.O. Box 155 

dmno, WI 54 963 
(AJ4) 685-5 707 



CHOICES 



CSG Corporation 

277 S. Washington SLreet 

Suite 209 

Alexandria, VA '>2314 
(703) 684-1101 



riie i'oilo\.ring is a listing of the evaluatidri forirs acquired arid reviewed 
by project staff. Many of these forms aro copyrighted arid permissic : to 
reproduce them ts required from the developer. Persons wishirig to exaririe 
any of the forms may contact the developer at the address given or corisult 
the published source. Pubiished srorces cited here can be located in the 
References in this report. 



i t le 



Developer/ Source. 



Vocational Education 



V.vn 1 uat ion Form 



Indiana University of Pensylvariia 
College of home Economics 
Published source: Ho vis arid Bloom 
(198:^) 



Vo c a t i cMi a 1 Ed u c a t i o n So!" I wa r c 
Eval uat i on 



Oregon State Department of Education 
700 Pringie__Parkw:. y 5E - 
Salem, OR 97310 



Unt it 1 ed Form 



Gswegb Cbuiitv BOCES 

M ic ro c bmput e r Su pp o r t Se r v ic e 

County Route 6^ 

Mexico, NY 1311^ 



So fc ware "" -aluai: ion Form 



' i.M ocomput er [nf-.trnct Lonal 
o 1 1 va r c Evaluation 



Published source: The Use — of 
MicracoTflpdt ers in Vbcat i.aria l 
Agriculture (Mincemdyer .1^83.- 

Published source: Mj^ri^compu t e r s* ? 
Voc at iorial- Traininp for -- -Small . 
Business ^aria.'^ement (Heath and 
Camp 1983) 



Microcomputer Business Software 
Kvalu ir i on - 



Cien^ira/ Educai: 



ion 



■ublished source: Mic rocomput e r b 
V^ catidn ^ -Training for Small 
■B^j ^n-e-s-r Management (Heath and 
Camp I98ji) 



Cj:»ur3e':are Evaluat ibri arid 
Descriptibri 



Mi era SIFT 
Published sofrce: 
(1983) 



Jones ^and Vaughan 



M i \ L-bcumputec CoiirsevNfare 
Evi.i J ua L i c)i Fb r;: 



hPIE/CU 

Pabi ish<='d source : 
(1983) 



Jones and "^^aughar 
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Sciu)*ii Mlc rowjrc: Eva.luatibri Form 



Dresden Ai^Bociates 

Published source: Jones and Vaughah 
(1983) 



Courseware Report Card 



Educational Insights, Inc. 
Pabiished source: 'ones^arid Vaughari 
. (1983) 



MlcrdcOTTipUter Educational 
Materials Evaluation 



MECC . . 

Published source Jones arid VcCugHari 
(19 83) 



ChecklisL for Microcomputer ' 
Program Revision 



SOFTSWAP 

Published source: Jones arid Vaughan 
(1983) 



}• v<j:l:ua ti on Form for Micro- 
coTtipu ter-Based Instructional 
Ma teri als 

Software Eva 1 via t ion Checklist 



CONDUIT 

Published source: Jones anH Vaughan 
(1983) 

Natibrial Couricii of Teachers of 

Mathematics 
Published source: Jones and Vaughan 

(1983) 



Sonolas tic Sof ^^are Evaluation 



So. twarc Evaluation Form 



Scholastic Book Services 
Published sburco: Jones and Vaughan 
(1983) 

E lec^r bnl^- -T^ .-^rnj jig - 

Published ;;oUrcK^. : Jones and Vaughan 
(1983) 



Computer Software Evaluatibri 
Form 



Alexandria City Public Schools 
Published source: Dearborn 
(1982) 



Software Evaluation FDrm and 
Checklis t 



The Computing Teacher 
Published source; Hilgenfeld 
(1982) 



Evaluatibri of Coursew^ 



Nancy Lee Olsen 
800 Biyant A", ^ . 
Worthing "on , Ohio 43085 



>.ir.-:wrire Kva'uation Fonn 



^Mcro computer Resources Ceriter 
TeacfiCi. : College 
Columbia University 
firoadw ly & 116th St. 
New Y rk, NY 1002 3 
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M 1 r o c o m p ti t e r instructional 
Software Evaluation 



initial Evaluation Form 
Teacher Evaluation Form 



I ns t rue t Ib ha J So f tt^g^ je -1-ej^jj^n 
PublisHed source: UbUg.las ^ Neiglits 
(n.d.) 

Waukesha County Technical Institutci 
Waukesha, WI 



Microcomputer Software Evaluation: 

Educational Courseware 
Microcomputer Software Evaluation: 

A]>plication Software 



B. June Schmidt, Assistant Professor 
Division of Vocational and Technical 

Education 
Virginia Poly technic Institute and 

State University 
213 Lane Hall 
Blacksburg, VA z 061 



Courseware Evaluatio;i (Revised) 



Mic-' >3IFT 

Not .uwest Regional Educational 

Laboratory 
300 S. W. sixth Avenue 
Portland, OR 9720A 



'omputer Program Review 



^ '"IJ^nHl %n R Teacher 
Published source: Watt 
(19 



Evaluation of CAI Cdursuware 



Random ffcuse, Inc . 
A 00 Hahn Road 
Westminster, >tD 21157 



Courseware Evaluation Worksheet 



College of Education 
University of Illinois at 

Ch amp a ± gn-U rb ana 
Publisht :! source: Dennis 

(19/9) 



Software- Evaluatir>n, Educational 
^ and Tccimicai 



CK Courseware Evaluation Form 



T e d d B r u mb ^3 . k ?:h 

Director of 'Research, ResburcL: arid 

Development _ ^ i- _ 
Mesa County •alley School District 51 
2115 Grand Avenue_. 
Grand JuncLiori, CO 81501 

I'ublished sc.'rce : "Curriculum Review 's 
Guidelines'' (M«y 1982) 



Kevluv Kau ? in, " >im 



f^'i.'.dc t(j th : Software Assessment 
i^rbcedure. Reviewer Dbcumerit 
Pi: Courseware 



CONDUIT 

Piibx J sh J source: Peters and Hepler 
(1982, 

nVJi Educational CDinputer Service 

Published source: National Education 
Association (1983;^) 



EKLC 
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CiiUk- to the Software Assessmenc NEA Educational Comnuter Setvice 

Procedure. Revit^wer DocamenC 1)2 : Published source ..National Education 
Applications Software Association (1983b) 

Guide to Che Software Assessment, .ND\_ F/ducatibrial Con^pUter Servic/ 

Procedure. Reviewer Document /?3: Published source: National Education 
CombLuation Products Assbciat idri (1983c) 

EPIE and eonsumers Union Micro- EPIE/CU 

Courseware PRO/FILE & P.O. Box 839_ 

Evaluation Water Mill, NY 11976 . 

Evaluation of Educational Arizona Center for Vocati5nal Education 

Course Ware Box 6025 

Flagstaff, AZ 8601 1 

Signer Studen-t Software The Computing Tea chc^r 

. (^lostionnaire; and Signer Publ ish:-d .source : Signer (1983) 

teacher Software Evaluation' 

Chocklisr. 

CAI Materials Appraisal ^^rocess^ The Ohio 3tate University 

College of Medicine 
Division of Computing Services for 

Medical Education and Rese^irch 
076 Health Sciences Library 
Colambus, OH 43210 



, .1 •at-ioh-CAI Mater-J ^ls 
Appraisal P voces. 



The Ohio State University 

College of Medicine 

Division of Computing Service^- for 

Medical FJucation and Resep/ rli 
076 Health Sciences Library 
Columbus i OH 43210 



Kvaluatibn Form 



EMC I ublishing Co. 
300 York Avenue _ 
St. Paul , Ha 5 5iUl 



Software Evaluation Checklist 



Francis Tuttlo Vb-Tech Center 
12777 North Rockwell 
aahoma City, OK 73142 



Checklist for Identifying Learn- 
ing Progr:^ -hpracteristics 

Modified MCE Pro; ram F.valnatibn 
Fo rm 



E4u^na^ -Te chno 1 O Ry 

Pubi.'shed source: Wado (1980) 

S5 ffl tiers i n Spe cial_Ed . - c_a_t ion 
PUb:i <1 source: Tat • r (198j; 
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